Summary
HotpotQA is a 113k-question Wikipedia QA dataset built so that answering requires reasoning over two documents rather than matching a single sentence. Crowd workers were shown pairs of paragraphs linked by a “bridge entity” in Wikipedia’s hyperlink graph (or two entities from the same curated list, for comparison questions) and asked to write a question that needs both, plus the sentence-level supporting facts. Questions are free-form text rather than derived from a knowledge-base schema, and the supporting facts give strong supervision for explainable reasoning. Baseline models do far worse than on SQuAD, especially in the open-domain “fullwiki” setting, which is why it became the standard multi-hop retrieval benchmark.
Key points
- distractor setting: has the gold paragraphs inside the context
- fullwiki setting: given everything, the hardest setting
- Two settings: distractor gives the two gold paragraphs plus 8 TF-IDF-retrieved distractors; fullwiki gives only the question and the first paragraphs of all of Wikipedia, so retrieval is part of the task.
- Collection: hyperlink graph over Wikipedia first paragraphs, bridge entities restricted to a curated page set; 42 curated entity lists for comparison questions (including yes/no); ~113k pairs.
- Supporting facts (sentence-level) are labeled and evaluated separately, so a system is scored on answer EM/F1, supporting-fact EM/F1, and a joint metric.
- Baseline (BiDAF-style + supporting-fact supervision) test scores: distractor 45.5 EM / 59.0 F1 on answers; fullwiki 25.2 EM / 34.4 F1 — retrieval is the bottleneck. Supporting-fact supervision improves answer accuracy.
- Question types: bridge-entity, comparison, and a share of questions that need the second hop only to locate the answer paragraph.
Relevance
- Retrieval Benchmarks — the canonical multi-hop dataset; its item shape (question, answer, supporting sentences, source paragraphs) matches the “question, answer, suggested answers, source nodes” a retrieval benchmark item needs, and the distractor/fullwiki split is a clean way to separate reader quality from retriever quality.
- Used as the multi-hop testbed in Demonstrate-Search-Predict.
Notes
First looked at 2023-05-26.