TestHelix: Checking Agent-Written Tests Against Other Implementations, Not a Single Reference

STEP 1. Diverse synthesis: K isolated agents, each with its own strategy
       → each produces a pair (test T_k + implementation P_k)
       → condition: T_k fails on the original code and passes on P_k

STEP 2. Cross-execution: test T_i against the other agents' implementations P_j
       → list of conflicts (test too strict or implementation broken)

STEP 3. Semantic review: a separate reviewer sees the requirements and tests, but NOT the implementations
       → objections only with a quote from the requirements

STEP 4. Fixes + "differential challenges":
       if two implementations pass all tests, look for an input where they diverge
       → repeat steps 2–3 after the fixes

STEP 5. Merge compatible suites → verify in a clean environment

The steps run in separate sessions: the agents are isolated from each other, and the reviewer doesn't see the implementations. This is a manual process across several chats or agent sessions, not a single prompt.

Task: You're building a payout service for a freelance marketplace that calculates the platform fee. The fee is 4% for individual clients and 6% for business clients, with an annual earnings cap of $200,000 per freelancer. The agent produced "green" tests, but you can't rely on them: they were checked against a single implementation. You need a suite that will catch cent rounding, exceeding the cap, and mixed-up rates.

Prompt 1 (run in 3 separate sessions, changing only the strategy):

You are agent #1. Exploration strategy: "boundary values and money" 
(cents, rounding, the cap, zero amounts).

Requirement (the single source of truth):
The function calc_fee(income_usd, client_type, year_total_usd) calculates the platform fee. 
Rate: 4% for client_type="person", 6% for client_type="company". 
Annual earnings cap is $200,000: if year_total_usd + income_usd exceeds the cap, 
the function raises LimitExceededError. The result is rounded to the cent.

Do two things independently:
1. Write tests T1.
2. Write your own implementation P1.
Condition: T1 must FAIL on an empty stub and PASS on P1. 
Run them and confirm. Don't tailor the tests to details of P1 that aren't in the requirement.
Output: the test file, the implementation file, the run log.

In the other two sessions the strategies are: "client types and errors" and "input combinations and argument order."

Prompt 2 (cross-execution):

Here are tests T1 and implementations P2, P3. Run T1 against each.
For each failure, determine what's to blame: 
(a) the test is too strict (demands something not in the requirement), 
(b) the implementation actually violates the requirement. 
For each verdict, quote the line from the requirement.

Prompt 3 (reviewer with no access to the implementations):

You are the reviewer. You have only the requirement and the tests. You can't see the implementations.
Check every assertion against the requirement. 
A blocking objection is allowed only with a quote from the requirement.
Separately list the behaviors from the requirement that NO test checks.

Result: You'll get three test suites, a "which test failed on which implementation" table, and a list of disputed spots with quotes from the requirement. The reviewer will surface untested behaviors: for example, the boundary at exactly $200,000 or mixed client types. After fixes and a rerun, you'll have a merged suite verified against several independent implementations. The specific findings depend on your code.

The weakness: the agent writes the test and the implementation in one head, and the test bends to fit the known solution. The "failed on the old, passed on the reference" check only catches gross failures. In the failure analysis, 50% of cases are weak or wrong checks, and another 34.6% are missed input combinations. A reference shows neither.

The strength: an LLM is good at writing different solutions to the same task if you give it different strategies. It's good at matching text against text: an assertion against a requirement, as long as it isn't tempted to peek at the code. Execution is a deterministic judge: a test either passes on someone else's implementation or it doesn't.

How the method exploits this: the other implementations act as a "jury panel." A test that rejects a correct but differently written version is too strict. A test that lets a broken one through is too lenient. A reviewer with no access to the code can't fit the verdict to the implementation.

Control levers: - Number of pairs K (3 and 5 in the paper) → more diversity, but pricier. K=5 gave a small gain over K=3. - Exploration strategies → vary them by task type: boundary values, errors, input combinations, performance. - The "objection only with a quote from the requirement" rule → cuts off the reviewer's made-up nitpicks. - Diversity of the implementations themselves → the more they differ, the more test errors get exposed.

The paper puts the full prompts in an appendix, and they're not in the text. The template below is assembled from the method description and preserves its structure: isolation, internal consistency of each pair, cross-execution, review without the implementation, differential challenge.

=== ROLE A: GENERATOR (run in K separate sessions) ===
<Role>You are agent #{number}. Exploration strategy: {strategy}.</Role>
<Requirement>{requirement_and_public_specs}</Requirement>
<Task>
1. Write tests T_{number}.
2. Independently write implementation P_{number}.
3. InternalConsistency:
   While (T doesn't fail on the original code OR T doesn't pass on P OR budget {iteration_count} not exhausted):
       run, look at the output, fix the pair.
   Don't put details into the tests that aren't in <Requirement>.
</Task>
<Output>tests, implementation, run log</Output>

=== ROLE B: CROSS-EXECUTION ===
<Input>tests {T_i}; implementations {P_j, j≠i}</Input>
<Task>
For each failure of T_i on P_j determine: 
  (a) the test is too strict, 
  (b) the implementation violates the requirement. 
Each verdict comes with a quote from <Requirement>.
</Task>

=== ROLE C: REVIEWER (does not see implementations) ===
<Input>{requirement}; tests {T_i}</Input>
<Rules>
- Check every assertion against the requirement.
- A blocking objection only with public evidence (a quote).
- List the behaviors from the requirement not covered by any test.
</Rules>

=== ROLE D: DIFFERENTIAL CHALLENGE ===
<Condition>Two different implementations pass all approved tests.</Condition>
<Task>
Find inputs on which they produce different results.
Decide from the requirement which result is correct, and add a test.
</Task>

=== FINISH ===
Fix the tests per the objections → repeat roles B and C → 
merge compatible suites → run the merged suite in a clean environment.

What to fill in: {requirement_and_public_specs} — the task text or documentation available to everyone. {strategy} — the angle of attack on the tests. {iteration_count} — the attempt budget per pair, e.g. 5.

🚀 Quick start — paste into a chat or an agent:

Here's the TestHelix template. Adapt it to my task: {describe the function or module 
that needs tests}. Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the requirement, the language and how the tests are run, and suitable exploration strategies. Without a clear requirement, the reviewer has nothing to stand on. Without a run command, cross-execution is impossible. From the template it takes the structure: isolated roles, the consistency loop, review without access to the implementation.

⚠️ Cost: the method needs several independent agent sessions and repeated runs. The authors compared variants under the same compute budget, but in your work that's time and tokens. Overkill for a one-off simple function.

⚠️ You need a clear requirement: the reviewer matches assertions against the requirement. If it's vague, there won't be any "with a quote" objections, and the review turns into arguments about taste.

⚠️ You need a runnable environment: the method relies on real test runs. Without an environment to run them in, the scheme doesn't work.

⚠️ Reliability is still not high: even with the method, the share of fully reliable suites stays far from 100%. It improves the odds but doesn't guarantee anything.

⚠️ Part of the method isn't available to you: the recursive self-improvement (RSI) stage needs hundreds of training tasks and a special optimization tool. For a practitioner only the core is realistic: synthesis, cross-checking, review.

⚠️ The effect is confirmed on two models, and the full prompts are in an appendix that's missing from the available text.

The idea was simple: change the yardstick. Tests are usually evaluated against a single reference solution. The researchers collected 300 tasks from 17 sources (six languages, 67 projects). For each one they assembled a panel of 10 implementations: the reference, 4 other correct ones, and 5 incorrect ones. Different agents wrote the implementations, and before being added to the panel they were checked for structural diversity of the code. The new metric (Joint Success) counts a task only if the tests fail on the original code, accept all correct implementations, and reject all incorrect ones.

Then they ran 14 coding-agent configurations from six model families. The best scored 28% on the new metric and 59.67% on the old one. So the old yardstick overstated quality by about 2x. Code coverage correlated with success (0.91 and 0.96), but doesn't guarantee correct checks on its own.

To understand exactly what's wrong with the tests, experts manually analyzed 286 failed runs. 50% were broken checks, 34.6% missed input combinations, 15.4% errors in executing the test itself. A separate experiment: as the panel grows from 1 to 15 implementations, the success rate drops, for example from 49% to 24% when correct and incorrect ones alternate. The more distinct solutions, the more hidden defects come to light.

For TestHelix, they ran ablations on two models. A single test+implementation pair gave about 20–22%. Three pairs without cross-checking barely improved the result (21.00 and 22.67). With cross-checking, the result rose to 27.67 and 29.00. Five pairs added a little more. Adding RSI lifts the metric 8.67–9.00 pp above the agent's "native" harness. In the truncated part of the text the full RSI numbers aren't visible. The key practical insight: diverse implementations are only valuable when the tests are actually run against the other solutions.

🔧 Technique: lock the rule into the agent's instructions file → the agent itself stops treating a "green test" as done

## Rules for writing tests
- A test is not done if it has only been checked against one implementation.
- Before finishing: write 3 alternative correct implementations with a different code structure 
  and 3 plausibly wrong ones (a boundary bug, data loss, the wrong branch). 
  Run the tests on all six.
- For each failure on a correct implementation, state which requirement doesn't allow 
  such a strict check. For each pass on a wrong one, state which assertion is weak.
- An assertion that checks only a count (len, count) without the contents is suspicious. Strengthen it.

This is a simplified version of the method: instead of separate agents, a single agent writes the "panel" itself. This version wasn't tested in the paper. Isolating the generator and the reviewer is part of the method, and it's lost in a single context. For important modules, it's better to split the roles across sessions.

• Paper: TestPrism: Rethinking Test Evaluation Beyond a Single Reference
• Authors: Han Li, Lingxiang Hu, Jiacheng Huang, Ziqian Jiang, Jingkai Luo, Wei Gao, Yunfan Tan, Zun Wang, Jiaheng Liu
• Affiliations: Nanjing University, Hong Kong Polytechnic University, Tencent
• Sources and tools mentioned: SWE-bench, Terminal Bench, SWT-Bench, TestEval, AutoSaddler (strategy optimization), Harbor (common task interface)