Summary
DSP is a framework for composing a frozen language model (LM) and a frozen retrieval model (RM) into task-aware programs instead of the fixed “retrieve-then-read” pipeline. A program has three stages: Demonstrate bootstraps demonstrations for every intermediate step from end-task labels alone (weak supervision), Search has the LM generate and decompose queries across multiple retrieval hops, and Predict produces a grounded answer from several passages, optionally with self-consistency. With GPT-3.5 and ColBERTv2 and no fine-tuning, DSP programs beat the vanilla LM, retrieve-then-read, and self-ask on Open-SQuAD, HotPotQA, and QReCC. The paper is the precursor to DSPy and argues that RAG pipelines should be written as short programs of composable transformations.
Key points
- Retrieve-then-read fails when no single passage answers the question; multi-hop questions need the LM to write intermediate queries and the RM to be called more than once.
- Demonstrate stage: given only question-answer pairs, run the pipeline on training examples and keep the traces that reach the right answer as demonstrations for every intermediate transformation. No hand labels for queries or hops. Selection primitives:
sample,knn,crossval. - Search stage: query rewriting for conversational context, iterative decomposition with summarization of hops; Predict stage: answer from multiple passages, majority vote (self-consistency).
- Results (Table 1, EM/F1): HotPotQA 51.4/62.9 vs 36.9/46.1 retrieve-then-read vs 28.3/36.4 vanilla LM; Open-SQuAD 36.6/49.0; QReCC F1 35.0. Relative EM gains of 37–120% over the vanilla LM and 8–39% over retrieve-then-read.
- Evaluated on Open-SQuAD, HotPotQA (fullwiki-style, open domain), and QReCC; these are the datasets a RAG benchmark would want to cover.
- Datasets used: Open-SQuAD, HotPotQA, QReCC (fills in the blank from the original note).
Relevance
- Retrieval Benchmarks — shows what a strong RAG pipeline looks like beyond retrieve-then-read, and what evaluation settings (open-domain, multi-hop, conversational) and metrics (EM, F1, novel-F1) were used to compare pipelines. A benchmark for retrieval pipelines needs to distinguish retrieval failures from LM failures, which DSP’s stage-wise structure makes visible.
- Uses HotpotQA as its multi-hop testbed.
Notes
First looked at 2023-05-26. Datasets used: (left blank at the time).