TRACER: reviewing an agent's code through its development history, plus a "memory" of fixes

PHASE A — reviewing the agent's patch
STEP 1: Find the changed functions and classes (not diff lines) → list of symbols
STEP 2: Attach the agent's explanations from its log to each function → quote + verification plan
STEP 3: Check each claim against the CURRENT code → confirmed / refuted
STEP 4: Record the hypothesis in the decision log (JSON): function, input boundary, suspected flaw
STEP 5: Repair gate: confirmed assessment + check of the function's callers + functional tests
        → patch accepted / rejected

PHASE B — protecting the fix going forward
STEP 6: Write a note: invariant + reason + boundary test (stored separately from the main code)
STEP 7: Every new edit must pass: regular tests + new-feature tests + the boundary test
STEP 8: Failure → return the failed check to the agent, one attempt; if that fails → keep the protected code

All of this is one workflow inside the host agent. Separate prompts are needed only for phases A and B.

Task: An online store's Python team accepts payments via Stripe webhooks. Claude Code rewrote the webhook handler. Tests are green, but the agent noted in its log: "the signature is already verified in middleware." A month later a manager asks: "Support the legacy notifications from our partner; they don't have a signature." You need to review the patch and make sure that request doesn't open the handler up to forgeries.

Prompt:

You are a security reviewer for code written by an agent. Follow the process below.

Changed function: handle_payment_webhook (file payments/webhooks.py)
The agent's task: "Rewrite the Stripe webhook handler, add idempotency by payment_id"
From the agent's log (its explanations, not code):
  "The request signature is already verified in middleware, so in the handler
   it's enough to parse the JSON and update the order status."

1. Treat the agent's words as a HYPOTHESIS, not a fact. Write out the claim verbatim.
2. Draw up a verification plan: which functions and calling code you need to read
   to confirm that the signature really is verified BEFORE handle_payment_webhook
   on every call path.
3. Check against the current code. Verdict: confirmed / refuted / insufficient data.
4. If refuted, propose a minimal patch. Accept it only if
   (a) the assessment is confirmed by the code, (b) every call site of the function has been checked,
   (c) functional tests pass.
5. After the fix, write a note: the invariant ("the handler never runs without a valid signature"),
   the reason, and a boundary test that sends a request without a signature and expects a rejection.
6. For any later edit to this file: run this test. If it fails,
   don't change the invariant; extend the solution within it.

Result: The model writes out the agent's quote and a verification plan. Then it walks the call chain, and for each path you'll see whether the signature is checked there. If there's a hole, you get a minimal patch marked with the checks it passed. At the end comes a note with the invariant and the boundary test code. When the "legacy partners without a signature" request arrives later, that note and test won't let anyone simply switch off verification. The model will have to propose another route, such as a separate locked-down endpoint or an IP allowlist.

The weakness. Scanners and tests see only the final code. Code with a missing check often looks fine and sails through every test. A reviewing model with no hint doesn't know where to look either: the repo is huge, with hundreds of functions.

The strength. The agent left a history: what it changed and what it said along the way. Its explanations contain assumptions: "the input is already validated," "the calling code is trusted." Each assumption becomes a concrete verification task: open the validator and see who calls the function. A model reads code well and checks it against a claim when the claim is stated explicitly. On its own it's weaker at hunting for "what could possibly be wrong here."

How the method uses this. It narrows the search to the changed functions (the right function lands in review scope almost every time, not in 13% of cases). It turns the agent's words into verifiable commitments. And the boundary test, stored separately, remembers the reason for the constraint, which regular tests don't capture.

Control knobs: - What to give the reviewer. The agent's explanations worked best (configuration C2). Adding the function's change history (parse tree, AST) made results worse on both models. More context isn't better. - Number of attempts. The study allows one attempt after a failed check. More attempts cost more; fewer may leave no chance to fix things. - The repair gate. You can tighten it: require the boundary test to run before a patch is accepted, not just after. - Who owns the boundary test. It's stored separately so the agent can't rewrite it along with the code.

The paper has no ready prompt text, only a description of the process. The original skill lives in the authors' repository (link under Resources). The template below is a reconstruction from the process description, not a copy of the original. I kept the structure: decision log, repair gate, invariant note, maintenance gate.

<Role>
You are a security reviewer for code written by an agent. Your job is to find
missing checks and protect fixes from future edits.
</Role>

<Input>
Changed functions: {function_list}
Agent's task: {task_description}
Agent's explanations from the log (text only, no code): {agent_explanations}
Scanner warnings (if any): {scanner_warnings}
Current code of the functions: {code}
</Input>

<ReviewLogic>
For each changed function:
  1. Write out the agent's claims about inputs, calling code, libraries.
  2. Every claim is a hypothesis. Draw up a verification plan: what to read.
  3. Check against the CURRENT code (not against the explanation!). Verdict:
     confirmed / refuted / insufficient data.
  4. If data is insufficient, propose running verification code on normal
     and boundary inputs.
  5. Add an entry to the decision log:
     {"hypothesis": ..., "function": ..., "input_boundary": ..., "suspected_flaw": ..., "verdict": ...}
</ReviewLogic>

<RepairGate>
Accept a patch only if ALL conditions are met:
  - the flaw assessment is confirmed by the current code;
  - every function that calls the one being fixed has been checked;
  - functional tests pass.
Otherwise leave the code unchanged and explain what was missing.
</RepairGate>

<RepairNote>
After an accepted patch, create a note:
  - invariant: {what_must_stay_true}
  - reason: {why_the_constraint_is_needed}
  - boundary_test: an executable check that fails if the invariant is violated.
Store the test separately from the main code.
</RepairNote>

<MaintenanceGate>
On any later edit to these functions:
  1. Run the regular tests, the new-feature tests and the boundary test.
  2. If the boundary test fails, show the failed check and allow ONE attempt
     to rework the implementation without weakening the invariant.
  3. If that fails, keep the previous protected code and report it.
</MaintenanceGate>

What to fill in: the list of functions from the diff, the task text, the agent's verbatim explanations (from the log, minus code blocks), scanner results if any, and the code itself. The key field is the agent's explanations: without them the method loses its core.

🚀 Quick start — paste into the chat:

Here is the TRACER template for reviewing an agent's code. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which functions the agent changed, where its explanations live, what the function does and who calls it. Without that it can't check the agent's claims against the code. Then it will write the invariant and boundary test for your project on its own.

⚠️ The review effect is small: The best configuration (code + agent explanations) produced about 5 extra fixes per 345 patches on the main test. The confidence interval includes zero. The authors themselves say it's a consistent ordering across two models, not a big gain.

⚠️ More context makes it worse: Adding the function's change history on top of the agent's explanations reduced the number of fixes on both models. Don't bury the reviewer in everything you've got. The authors didn't pin down the mechanism: "context dilution" is just a hypothesis.

⚠️ The agent's explanations are unreliable: They don't reflect the model's actual reasoning. They're only hypotheses, can't be taken on faith, and the method is built on checking them.

⚠️ You need access to the agent's log: Without explanations and edit history, the method loses its main idea. It doesn't work for code written by humans.

⚠️ No gain for already-protected code: Where protection was already in place, the agent's explanations added almost nothing (0 and 2 fixes out of 110).

⚠️ A lot gets lost along the way: For one of the models, only a small fraction of cases reach a confirmed assessment (15–32 of 387). I don't see a full breakdown of the reasons in the text; it's truncated. The method doesn't catch everything.

⚠️ Narrow testing: Experiments were on Python projects only. The second study (fix memory) covers 5 components and 10 tasks — a small sample. A single type of protection (the boundary test) doesn't cover every kind of vulnerability.

The researchers used the public SusVibes benchmark: 186 tasks in real Python repositories where an agent writes code, and functionality and security are checked separately. Of 1302 patches from seven "author" agents (Claude 4 Sonnet, GLM-5, Kimi K2 and others), 387 remained after filtering. Two reviewer models, DeepSeek-V3.2 and Gemini-3.8-Flash, tried to fix the vulnerabilities in these patches under four configurations. The baseline saw the code and scanner warnings. The others added the function's change history, the agent's explanations, or both. That came to 3096 runs in total. Hidden security tests were used only for scoring and never shown to the reviewers.

First they checked whether the right function landed in review scope: tracking changed functions gave 94.6% versus 13.4% for the scanner's primary warning. Then they counted verified fixes. The agent's explanations gave the best result on both models: 76 and 75 passing security tests versus the original 68. Adding AST history lowered the result. Functional regressions were rare, at most one case. The comparison with the baseline isn't clean: C1–C3 also get "audit hints," so what's being compared is data bundles, not individual channels.

The second experiment tested memory. Five components of structural-biology software (industry partner Nostrum Biodiscovery) were first fixed, then run through 10 feature-addition tasks. A 2×2 design: with or without a fix note, with or without a boundary test. With functional-test feedback alone, 4 of 10 tasks succeeded with 3 protection violations. With a note and a boundary test, 9 of 10 succeeded with zero violations. The surprise is that functional tests themselves nudge the model toward weakening protection: for the sake of "compatibility" it brought back weights_only=False, and every test passed.

🔧 Technique: add a "boundary test" to the agent's instruction file → protection against rollbacks

In CLAUDE.md or AGENTS.md:

Copy

Before editing any file on the protected list ({file_list}), read
SECURITY_NOTES.md. For every invariant listed there, run its check
after the edit. If a check fails, don't weaken the invariant; find another way.

This is my adaptation of the invariant-note idea; the paper has no such file. The notes live in a plain file that the agent reads on every edit. Test it on your own project.

🔧 Extrapolation: the agent reviews its own PR before submitting

Copy

Before submitting the PR, write out every claim you made about input data
("this is already validated," "the calling code is trusted"). For each one, find
the place in the code that guarantees it. If you can't find it, add a check or
explicitly flag the assumption in the PR description.

This transfers the "an explanation is a hypothesis" principle to the agent's self-check. In the paper the check is done by a separate reviewer, not the author itself, and the authors didn't test self-review.

• Paper: Securing Agent-Generated Code with Its Own Development History
• Authors: Prateek Kumar Rajput, Shahzeb Khan, El-Hacen Diallo, Yewei Song, Jacques Klein, Tegawendé F. Bissyandé. University of Luxembourg, Nostrum Biodiscovery S.L. (Barcelona).
• TRACER skill: https://github.com/pkrajput/TRACER_skill
• Benchmark: SusVibes (evaluates functionality and security with separate test suites)
• Related: ReAct, SWE-agent, CodeQL, Semgrep, GumTree, PAGENT