TRACE: diagnosing a drop in an agent's score — is it the agent or the measurement?

STEP 0: Controls → (a) repeat with no changes = noise baseline
                   (b) deliberately harmful change = “can the check see an effect”
STEP 1: Paired runs → the same tasks, before and after the change (several times)
STEP 2: Compare scores → difference, pass rate, comparison with repeat noise
STEP 3: Review trajectories → did behavior/outcome change or not?
STEP 4: Re-score → same trajectories, corrected rule
        gap gone → the checking rule is to blame
        gap remains → the agent's behavior is to blame

Each step is a separate operation on the data. This isn't a single prompt but a process of several requests or manual checks.

Task: A seller's team on a marketplace (say, Amazon) launched an agent to handle returns. It's tested on 20 test tickets, and the “pass/fail” score is set automatically. A developer renamed the tools to clearer names: get_order became fetch_order_details, refund became issue_refund_to_customer. The success rate dropped from 80% to 65%. The boss is already asking to roll back the prompt.

Prompt (sent to the chat along with the exported logs):

<Role>
You are an agent quality analyst. Your task is to figure out what caused the score drop: 
a change in the agent's behavior or quirks of the check itself. 
Don't draw conclusions from a single number.
</Role>

<Context>
Agent: handling returns on a marketplace.
What changed: the tools were renamed (get_order → fetch_order_details, 
refund → issue_refund_to_customer). The tools' logic and arguments didn't change.
Check: an automated script that outputs “pass/fail”.
</Context>

<Data>
Table A: 20 tasks × 3 runs BEFORE the change (result + action trace).
Table B: 20 tasks × 3 runs AFTER the change (result + action trace).
Table C: the same 20 tasks, a repeat run BEFORE the change with no edits.
[paste the tables]
</Data>

<Protocol>
Step 1. Noise. Using tables A and C, compute the share of tasks whose 
“pass/fail” result differs between two identical runs.
Step 2. Difference. Compute the change in success rate between A and B. 
Compare it with the noise from step 1. Say whether the difference exceeds the noise or not.
Step 3. Behavior. For tasks whose result changed, compare the traces 
before and after: same calls, same arguments, same outcome for the customer? 
Split the tasks into three groups: “same behavior”, “behavior changed”, “unclear”.
Step 4. Suspect the check. For the “same behavior, but score dropped” group, 
describe which checking rule might have fired differently (for example, matching by tool name). 
Propose a corrected rule and show how the score would change 
when re-scored on the same traces.
Step 5. Conclusion. Split the drop into two parts: 
how much the check explains, how much the actual behavior explains. 
State what additional control is needed to claim this with confidence.
</Protocol>

Result: The model works through the steps. First it estimates the background noise: how much the repeats themselves diverge. Then it compares that with the drop and says whether the drop goes beyond the noise. Next it sorts tasks into behavior groups and points to where the checking rule looks suspicious. At the end, the model splits the drop into “the check is to blame” and “the behavior changed”, with caveats where data is thin.

The weakness: a score is always “agent + environment + checker”. When the score changes, it's unclear who's to blame. On top of that, we tend to read any change as a change in capability. A simple check based on names, format or a template pretends to measure the outcome but actually measures the form.

The strength: a trajectory is evidence. If the agent has the same actions and the same outcome but a different score, the check is to blame. You can show this without re-running anything. Re-scoring on the same records is cheap and unambiguous: if the gap disappears after fixing the rule, you've found the cause.

Controls make the conclusion meaningful. A repeat with no changes shows that 15–36% of outcomes change on their own. Without that baseline, any “dropped by 5 points” could be noise. A positive control is needed for the opposite situation. If you made a deliberately harmful change and the score didn't react, your check is blind, and the conclusion “nothing changed” is worthless.

Control knobs: - Number of repeats — more repeats, more precise noise estimate. For a quick check, three per task is enough. - The “pass” threshold — vary it and see whether the conclusion holds at 0.55, 0.65 and 0.75. - Equivalence margin — the authors used ±0.10. For critical tasks, go narrower. - The suspected rule — plug in your own (date format, case, call order), and re-scoring will show its contribution.

<Role>
You are an analyst of agent quality (or of LLM-based automation). 
Your task is to separate the causes of a score change: the agent's behavior or the checking rule. 
No conclusions from a single number.
</Role>

<Context>
System: {what_agent_or_process}
What changed: {change}
How it's scored: {checking_rule_or_judge}
What was expected: {should_the_change_affect_the_outcome_or_not}
</Context>

<Data>
Before the change: {data_before}
After the change: {data_after}
Repeat run with no changes: {repeat_data}
Action traces (if available): {traces}
</Data>

<Protocol>
Step 1. Compute the noise: the share of tasks where two identical runs gave different results.
Step 2. Compute the before/after score change and compare it with the noise.
Step 3. For tasks with a changed result, compare traces: calls, arguments, outcome for the user.
Step 4. For “same behavior, different score” cases, name the checking rule that might have fired differently. 
       Propose a corrected rule and re-score on the same traces.
Step 5. Split the drop: share explained by the check / share explained by behavior. 
       State what data is missing.
</Protocol>

<Rules>
- Compute the “average score” alongside the pass rate: a small gap can mean failure in both cases.
- Don't call a cause proven if the difference doesn't exceed the noise.
- If there's too little data, say so plainly.
</Rules>

What to fill in: {change} — one specific change at a time (tool names, output format, task wording). {checking_rule_or_judge} — how exactly the score is set: a script, a checklist, an LLM judge. {data_*} — exported results as a table. {traces} — the agent's action logs.

🚀 Quick start — paste into the chat:

Here's a template for diagnosing a score change (TRACE). Adapt it to my task: [describe your agent/automation and what changed]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what exactly you changed, how the check works and what data you have. The method needs this to separate the culprits: it has to know what checks the outcome and whether there are action records for re-scoring. The model will take the structure from the template and adapt it to your system.

⚠️ You need run data: without several repeats and action records (traces), you'll only get a hypothesis. Noise and re-scoring require the actual materials, not a memory of “it was roughly like this”.

⚠️ Few tasks — wide intervals: on 30 tasks in the first study, almost all intervals included zero. The absence of an effect there didn't mean there was none: there just wasn't enough data. The only “clear” effect vanished when the sample was expanded.

⚠️ Tested on a narrow set: most of the experiments are one public benchmark (retail and airline), four agents and a synthetic environment. Conclusions like “renaming is safe” can't be carried over to any tools and prompts.

⚠️ A caveat about the checks themselves: the “matching by name” defect was found in a simple homemade script. It's not a conclusion about trained evaluator models.

⚠️ Misleading names do hurt: if a tool's name contradicts what it does, reward drops for every agent. Neutral renames are harmless; misleading ones are not.

⚠️ A judge can't be tested on stability alone: repeatability isn't validity. A judge can consistently grade the wrong thing (for example, the order of steps instead of the outcome).

The idea was simple: to trust the diagnosis, you first need to test it where the cause is known in advance. The author built a synthetic world of 25 research tasks (for example, fix the learning rate or prevent data leakage) and three “actor” agents with scripted behavior: a careful one, an “answer-memorizer” and a cheater. The same tool rename produced different results. For the careful agent the score dropped by 0.25, but after mapping the names back the gap went to zero — meaning the check was to blame. For the “memorizer” the gap remained (0.13): it really was producing a failed report. The cheater's gap was nearly zero (0.006). But that's not “robustness”: it failed both variants equally badly.

Then they moved to real LLM agents on the public τ²-bench: four agents, 30 tasks, three conditions (original, renaming, reformatting the answers). They saw an “effect”: Laguna's score rose by 0.267 with renaming. The authors didn't buy it and re-checked on 88 new tasks with three runs each. The effect vanished, and in 7 of 8 pairs the difference fell within ±0.10. To make the zeros meaningful, they added a positive control: the tool names were deliberately shuffled, but the tasks stayed solvable. Reward dropped for every agent by 0.20–0.44, so the setup can see an effect.

The biggest surprise was the noise. An identical repeat changed the result for 14.7–16.3% of tasks, and for 35.6% with Laguna. That means any single-run comparisons on benchmarks like these are unreliable. The judges (GPT-6.1 Sol and Claude Opus 5.5) were robust to presentation format on 118 fixed trajectories: the difference stayed within their own noise. But they disagreed with each other on 57% of records. Claude agreed with the reference in 95% of cases, GPT in 38%, mostly nitpicking the procedure — for example, multiple tool calls in a single turn. The practical takeaway: control for noise, look at trajectories, and don't confuse stability with correctness.

🔧 Technique: add an outcome clause to the judge's rubric → less nitpicking about procedure

The authors didn't test this themselves, but their finding suggests the move. One judge penalized procedure even though the benchmark counted the task as a success. If you use an LLM judge, spell out explicitly what's being graded:

Grade only the final outcome for the user: 
whether the task is solved and whether the final data is correct. 
Don't lower the score for the order of actions, the number of tool calls 
or style if the outcome is correct. 
If the order of actions is critical (security, irreversible operations), flag it as a separate item.

This is a hypothesis; test it on your own sample: take 20 records, run them through the old and new rubric and compare with human grading.

Extrapolation: checking tool descriptions in the agent's instructions

The authors showed that misleading tool names hurt every agent. From this you can build a self-check for an instructions file (CLAUDE.md, AGENTS.md):

Before using a tool, check its name against its description. 
If the name and description disagree (for example, “delete” actually archives), 
stop and report it — don't choose by name.

This is a conclusion inspired by the paper, not its result. The authors tested shuffled names specifically, but not an instruction like this.

• Paper: “TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation”, AI Measurement Science Workshop at COLM 2026.
• Author: Radhika Gaonkar, Prime Intellect.
• Benchmarks and environments: τ²-bench (Barres et al., 2026), ResearchOpsEnv (a synthetic environment of 25 tasks created by the authors).
• Related work: Krakovna et al. (2020) on specification gaming, Gao et al. (2023) on reward model overoptimization, Zhu et al. (2025) on auditing agent benchmarks, Ribeiro et al. (2020) on behavioral testing.