Vibrant Labs Research

More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering

2 min read

Summary

A survey of 47 textual QA benchmark datasets organised by application scenario rather than annotation style. It splits the field into classical machine reading comprehension (18 datasets), four “novel MRC” settings — conversational QA, multi-hop QA, long-form QA, cross-language QA (13 datasets) — open-domain QA (11 datasets, including KILT-style aggregations), and commonsense QA (5). For each dataset it records construction method, size, answer type and the metrics used, then summarises eight evaluation metrics (accuracy, exact match, F1, ROUGE, BLEU, METEOR, HEQ and related) and how their use depends on answer type. It closes with trends: more reasoning-heavy, multi-document, conversational and multilingual benchmarks, and a shift from span extraction toward free-form answers that need better metrics.

Key points

  • Taxonomy by application scenario: classical MRC → novel MRC (conversational, multi-hop, long-form, cross-language) → open-domain QA → commonsense QA. Traditional annotation styles (cloze, multi-choice, span, free-form) are cross-cutting.
  • Classical MRC benchmarks are saturated (models beat humans on SQuAD-style tasks), which drove the novel MRC and ODQA datasets.
  • Open-domain QA section lists the datasets actually used to train/test retrieval-augmented systems (NQ, TriviaQA, WebQuestions, HotpotQA, KILT, etc.) and their metrics — a ready-made shortlist for a retrieval benchmark.
  • Metrics section: accuracy and EM for short answers, F1 for spans, ROUGE/BLEU/METEOR for free-form answers, HEQ for conversational QA; metric choice tracks answer type.
  • Trends: harder reasoning, multi-document evidence, unanswerable questions, dialogue context, multilinguality, and the need for metrics beyond lexical overlap for generative answers.

Relevance

  • Retrieval Benchmarks — the “datasets” half of what the concept page says a benchmark needs: a catalogue of QA datasets with their item shapes and standard metrics, from which to pick sources of (question, answer, source nodes) triples; complements HotpotQA and the metric papers.

Notes

(batch ingest; no notes yet)

Graph View

Backlinks