Summary
QA-based factual-consistency metrics check a summary by extracting candidate answers from it, generating questions about them, answering those questions from the source document, and comparing the answers. This paper decomposes that pipeline into four components — answer selection, question generation, question answering, answer-overlap scoring — plus an answerability filter, and tunes each against the SummaC benchmark of six factual-consistency datasets. The optimised metric, QAFactEval, improves over prior QA-based metrics by ~14% and beats the best entailment-based metric (SummaC-Conv), with question generation and answerability classification identified as the decisive components. QA-based and NLI-based signals are complementary: a small learned combination (QAFactEval-NLI) pushes the state of the art further.
Key points
- Four-component decomposition of QA metrics: (1) answer selection (noun phrases), (2) QG conditioned on answers, (3) QA over the source, (4) answer-overlap scoring; plus (5) filtering unanswerable questions.
- Prior QA metrics (FEQA, QuestEval, QAGS) disagreed with each other and with NLI metrics largely because of component and granularity choices, not the paradigm.
- Results: QAFactEval ≈ 75.7 balanced accuracy on SummaC vs SummaC-Conv 74.4; +14% over previous QA-based metrics; a learned answer-overlap scorer (LERC) and answerability filtering matter most.
- Combining QA and entailment scores through a simple learned network gives a further boost; the two paradigms catch different errors.
- Reports 10 additional metrics with both classification and correlation analyses — the most complete comparison of consistency metrics at the time.
- Caveat: none of the metrics catch all inconsistencies; English/news only.
Relevance
- Retrieval Benchmarks — a faithfulness metric for the “suite of metrics”: QA-based consistency checks are the same mechanism a RAG benchmark would use to test whether an answer is grounded in the retrieved passages, and the component analysis is a checklist for building such a metric. Pairs with SummaC (NLI side) and BERTScore (similarity side).
Notes
(batch ingest; no notes yet)