Summary
BERTScore scores a candidate text against a reference by greedily matching each token to its most similar token in the other text using cosine similarity of contextual BERT embeddings, yielding precision, recall and F1 (optionally idf-weighted and baseline-rescaled). Unlike BLEU/ROUGE/METEOR it handles paraphrase and long-range word-order changes because it matches meaning rather than surface n-grams. Across the outputs of 363 machine-translation and image-captioning systems it correlates better with human judgments than existing metrics at both system and segment level, is better for model selection, and is more robust on the adversarial paraphrase set PAWS. It is task-agnostic, needs no external resources, and supports 104 languages via multilingual BERT.
Key points
- Method: pairwise cosine similarity between contextual token embeddings; R_BERT = mean over reference tokens of max similarity, P_BERT symmetric, F_BERT their harmonic mean. Optional idf importance weighting and baseline rescaling for readability.
- Fixes two n-gram failure modes: paraphrases scored below unrelated strings, and cause/effect clause swaps barely penalised.
- Evaluation on 363 systems (WMT translation, COCO captioning): higher system- and segment-level correlation with humans than BLEU, METEOR, chrF, and task-specific SPICE; stronger model selection than BLEU.
- Robustness: on PAWS adversarial paraphrases BERTScore degrades far less than other metrics.
- No single configuration dominates; recommended F_BERT with RoBERTa-large for English, multilingual BERT elsewhere (less stable on low-resource languages).
- Differentiable, so usable as a training loss; later variants (MoverScore with EMD, YiSi with BERT) confirm the direction.
Relevance
- Retrieval Benchmarks — a candidate answer-quality metric for the “suite of metrics” a retrieval benchmark needs: measures semantic overlap between a generated answer and a reference without requiring exact match, but says nothing about faithfulness to retrieved context (see QAFactEval, SummaC).
Notes
(batch ingest; no notes yet)