Vibrant Labs Research

Retrieval Benchmarks

2 min read

What it is

How to benchmark a retrieval pipeline. Two parts: questions to ask people who build IR systems, and what a benchmark needs.

Questions for users

What are the problems

  • how do you benchmark your retrieval pipeline currently? what solutions have you tried?

How intense is the problem

  • on a scale of 1-10, how important is having a proper process for benchmarking for you?
  • on a scale of 1-10, how satisfied are you with your current solution?
  • how frequently do you make changes to your pipeline?

General

  • what is currently the most important problem you face building and maintaining your own IR system?
  • which ones do you wish someone else handled for you?

What a benchmark needs

  1. A suite of metrics. Most important, since it gives an understanding of the pipeline people can use.
  2. Datasets. Vaguer, because people can use any dataset they want. Each item needs only: question, answer, suggested answers, source nodes.

Key sources

  • Demonstrate-Search-Predict — RAG as composable LM+RM programs; multi-hop/conversational QA settings and EM/F1/nF1 metrics used to compare pipelines
  • HotpotQA — canonical multi-hop dataset; item shape (question, answer, supporting sentences, paragraphs) and distractor/fullwiki split separate reader from retriever
  • BERTScore — semantic similarity metric for answers vs references; no faithfulness signal
  • QAFactEval — QA-based faithfulness metric decomposed into answer selection, QG, QA, overlap, answerability
  • SummaC — NLI-based faithfulness metric at sentence granularity and the SummaC consistency benchmark
  • More Than Reading Comprehension — catalogue of QA datasets (MRC, multi-hop, conversational, open-domain) and which metrics fit which answer types
  • Holistic Evaluation of Language Models — the structural model for a benchmark: taxonomy, multi-metric by design, standardised adaptation, explicit gaps

To ingest

Metrics

Graph View

Backlinks