Question Decomposition and Evidence Chain Scoring for Complex RAG Retrieval Quality Evaluation on the FRAMES Benchmark

Abstract

Complex retrieval-augmented generation (RAG) systems can retrieve one highly relevant page while still missing the other pages required to support a multi-hop answer. This paper presents QD-ECS, an article-level evaluation procedure that combines deterministic question decomposition with evidence-chain scoring on the FRAMES benchmark. The evaluation covers all 824 FRAMES questions and a closed candidate pool of 2,480 normalized Wikipedia article identifiers derived from the benchmark links. Five retrieval settings are compared: Random, TF-IDF, BM25, BM25-Decomp, and Hybrid-Chain. Retrieval quality is measured with article Recall@k, ChainExact@k, step evidence coverage (SEC@k), NDCG@k, and MRR. Hybrid-Chain achieved the highest Recall@5 (0.499), SEC@5 (0.438), NDCG@5 (0.552), and MRR (0.824), whereas TF-IDF obtained the highest ChainExact@5 (0.180), exceeding Hybrid-Chain by 0.001. The central finding is that first-hit success substantially overstates evidence sufficiency: Hybrid-Chain recovered every required article in the top five for 147 of 824 questions, and its ChainExact@5 declined from 0.373 on two-article questions to 0.011 on questions requiring five or more articles. QD-ECS therefore separates early partial recovery from complete chain recovery and provides a compact diagnostic framework for evaluating multi-hop RAG retrieval before answer generation