Summary
Earlier attempts to use natural-language-inference models to detect factual inconsistencies in summaries failed (≈52% accuracy) because they fed a whole document as premise and a whole summary as hypothesis, while NLI models are trained on sentence pairs. SummaC fixes the granularity mismatch: split document and summary into sentences, score every (document sentence, summary sentence) pair with an NLI model, and aggregate. SummaC-ZS aggregates zero-shot with max-then-mean; SummaC-Conv learns a 1-D convolution over the histogram of entailment scores per summary sentence. The paper also standardises six existing datasets into the SummaC benchmark for binary consistency classification, where SummaC-Conv reaches 74.4% balanced accuracy, five points above the best prior method.
Key points
- Key insight: NLI works for inconsistency detection only at sentence granularity; document-level input lets the model be fooled by mostly-entailed summaries.
- Two aggregators: SummaC-ZS (max over document sentences, mean over summary sentences; interpretable) and SummaC-Conv (learned convolution over binned score distributions; stronger but less traceable).
- SummaC benchmark: six datasets (CoGenSumm, XSumFaith, Polytope, FactCC, SummEval, FRANK) unified to a binary label with balanced accuracy as the metric.
- Results: SummaC-Conv 74.4% balanced accuracy, SummaC-ZS 72.1%, vs ~69% for prior best (DAE, FEQA, QuestEval, FactCC-CLS baselines included).
- Consistency ≠ accuracy: a summary that adds true information not in the source is accurate but inconsistent; the task targets consistency with the source.
- Future work: multi-granularity/multi-hop aggregation, interpretability, domains beyond news.
Relevance
- Retrieval Benchmarks — the entailment-based route to a faithfulness metric and the benchmark that QAFactEval is measured on; the sentence-pair decomposition maps directly onto checking each answer sentence against retrieved chunks in a RAG system.
Notes
(batch ingest; no notes yet)