Vibrant Labs Research

Failure-driven Task Mining

3 min read

The paper

Every time we argue that synthetic data generation is the biggest bottleneck in AI today — around Tau2-Infinity, our tool-use benchmark, or at our Autoscaling RL Environments dinner in SF — more people join the conversation. Two counterpoints come up again and again about autonomously scaling post-training data:

  • How do you make sure you’re mining tasks the model genuinely can’t complete?
  • How do you write verifiers for non-verifiable tasks?

This session went into two recent papers focused on exactly those problems:

  • TRACE from the Scaling Intelligence Lab at Stanford
  • RIFT from Snorkel AI

Discussion

TRACE primarily focuses on capability-based task mining. The team TRACE:

  • runs a model on a large set of existing tasks;
  • labels each trajectory with the capabilities it required, using an LLM;
  • clusters the capability descriptions into categories, again with an LLM — for example, trajectories described as “searching for an item” end up together;
  • selects capabilities that show up in at least 10% of failed trajectories;
  • calculates the “contrastive gap”, a measure of how much more often the capability was missing in failed trajectories than in successful ones.

The contrastive gap is an improvement over regular error analysis, which only looks at failures, because it reduces the number of false-positive capabilities. It also filters out capabilities that fail almost everywhere, which often reflect task ambiguity or annotation noise rather than a trainable deficit TRACE — a way to tell model capability gaps apart from harness or environment problems.

TRACE led to significant gains over the base model on benchmarks like Tau2-Bench TRACE.

Most suggestions for the non-verifiable task problem involve an LLM-as-a-judge approach. RIFT covers what failure modes are likely to occur with LLM-as-a-judge rubrics, and how to surface them.

In RIFT, you run the same trajectory through the same rubric n times (“reward variance”) across multiple models (“inter-model agreement”); low variance and high agreement mean the rubric is good (RIFT v1).

Since most realistic environments, especially enterprise ones, have a mix of write-based and read-based tasks, the reward signal is heavily influenced by the reliability of the rubric.

What we took from it

  • Mine tasks from contrastive failures, not raw failures: a capability counts only if it separates failed from successful trajectories and covers a meaningful share of failures TRACE (Environment Generation).
  • Contrastive analysis is also a cheap way to tell model gaps from harness or environment bugs before spending compute on training TRACE.
  • Treat a rubric as an instrument to be tested before it grades anything: check its reliability across repeated runs and judges, and its content against what actually matters RIFT (Benchmark Validity).
  • Techniques like TRACE and RIFT are how tool-use environments become larger, more robust synthetic post-training datasets (WorldSmith).

Graph View

Backlinks