Summary
HELM (Stanford CRFM) is a framework and living benchmark for evaluating language models “holistically”: start from a top-down taxonomy of scenarios (task × domain × language) and metrics (desiderata), pick a subset for coverage and feasibility, and be explicit about what is missing. Version 1 measures 7 metric categories — accuracy, calibration, robustness, fairness, bias, toxicity, efficiency — on each of 16 core scenarios (87.5% of the pairs), plus 7 targeted evaluations over 26 more scenarios (knowledge, reasoning, memorisation/copyright, disinformation, and so on). It benchmarks 30 models (open, limited-access, closed) on all 42 scenarios under standardised few-shot prompting, raising the share of core scenarios on which prominent models have been evaluated from 17.9% to 96%, and reports 25 top-level findings with all prompts and completions released.
Key points
- Three principles: broad coverage with explicit incompleteness, multi-metric measurement for every scenario, and standardisation (same adaptation strategy and scenarios for every model).
- Scenarios cover 6 user-facing tasks including question answering, information retrieval (MS MARCO passage re-ranking), summarisation and toxicity detection, across domains and English varieties.
- Metrics beyond accuracy are first-class: calibration, robustness to perturbations, fairness across dialects/groups, demographic bias and stereotypes in generations, toxicity, and inference efficiency; trade-offs are exposed side by side.
- Findings include: instruction-tuned models lead on accuracy; accuracy and calibration/robustness/fairness are correlated but not identical; open models lag closed ones; prompt formatting materially changes results.
- A modular open toolkit (crfm/helm) for adding scenarios, models, metrics and prompting strategies; intended to be updated continuously.
- Explicit “what is missing” list: non-English languages, more trustworthiness metrics, neglected dialects, etc.
Relevance
- Retrieval Benchmarks — a model for the structure of a benchmark rather than a single metric: taxonomy first, multi-metric by design, standardised adaptation, and honest gaps. Its information-retrieval scenario and its treatment of QA metrics are directly reusable; the multi-metric table is what the concept page’s “suite of metrics” could look like.
- Benchmark Research — the reference design for a holistic benchmark.
Notes
(batch ingest; no notes yet)