TL;DR
FALCON is a scheme for checking code an agent wrote. One step (the attacker) writes probe scripts: they call the function under test with malicious inputs. The scripts run in a sandbox and produce an execution trace: what went in, what came back, which errors and side effects occurred. A second step (the judge) looks only at that trace and decides whether security was violated. The attacker does not write expected results, and the judge does not read the source.
The core pain: a model acting as judge reads the code and eyeballs whether it is safe. It gets it wrong, and the agent in the edit loop learns to please that judge instead of fixing vulnerabilities. The "it's more secure now" report climbs, while real security on hidden tests stays flat. The reason: judging code means mentally walking every execution path, while judging a trace means assessing one real run. The second question is far easier. The judge on traces hit about 88% accuracy versus about 61% on source code.
The method comes down to three moves: the attacker writes probes without verdicts, the system filters out "cheating" probes, and the judge scores the trace. Every violation found goes back to the agent as a counterexample (probe + trace + reason), and the agent fixes the code. This runs for up to five rounds. Full FALCON is code with a sandbox and 21 detectors. But the principle "separate attack from evaluation, judge by the run" can be given to an agent as an instruction.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Pick targets (changed functions/classes, not tests)
STEP 2: ATTACKER writes up to n deterministic probe scripts
→ no expected results and no verdicts
STEP 3: FAITHFULNESS CHECK: discard a probe if
• it redefines/replaces the target
• it performs the "sensitive operation" itself and then displays it
• the trace contains no call to the target
STEP 4: RUN the probes in a sandbox → execution trace
STEP 5: JUDGE reads the trace (3 tiers):
crash → rules per vulnerability type → LLM reads the trace if unclear
→ insecure / secure / uncertain
STEP 6: COUNTEREXAMPLE (probe + trace + reason) → back to the agent for a fix
→ repeat up to 5 rounds
In the original, steps 2–6 are code and infrastructure. In a chat you can reproduce them with a few requests. In an agent that can run commands (Claude Code, Cursor), with a single instruction.
Example in Action
Task: you lead a small team building a customer portal for an online school in Austin. Claude Code wrote a /download?file=... endpoint for downloading contracts and receipts. Tests are green, files download. But "works" doesn't mean "safe": a user can pass ../../etc/passwd or someone else's contract number.
Prompt:
You are reviewing code you just wrote. Work in two separate modes.
<Attacker>
Target: the download_document function in app/documents.py.
Write 5 deterministic Python scripts in the /tmp/probes/ folder.
Each one calls ONLY download_document (import it from the project)
with a malicious input: path traversal (../), absolute paths, someone else's contract_id,
empty and very long values, null bytes.
Rules:
- Do NOT write an expected result and do NOT give a verdict.
- The script only prints: what it passed in, what came back, which exceptions,
which files were opened.
- Do NOT redefine or replace download_document or any modules on the path to it.
- Do NOT read sensitive files yourself in the script body — the function must do that.
</Attacker>
<FaithfulnessCheck>
Before running, check each script: does it replace the target, or does it
do itself what the function is supposed to do? Delete bad scripts and rewrite them.
</FaithfulnessCheck>
<Verifier>
Run the scripts. Now do NOT look at the function's source code.
Judge only by the run output. For each probe give a verdict:
insecure (with a quote from the output as evidence) /
secure / uncertain (no data).
If no probe reached the target, that is "no evidence", not "secure".
</Verifier>
For each insecure, build a counterexample: script, output, reason.
Fix the code based on the counterexamples, then repeat the loop. Maximum 5 rounds.
Stop when no insecure remains, and show a round-by-round summary.
Result: the agent will create the probe files and show, round by round, which scripts were rejected and which verdicts came back, with quotes from the output. Path traversal will most likely produce a counterexample with a trace. Then comes the fix (for example, path normalization and a contract-owner check) and a rerun. At the end: a summary of how many rounds, what got fixed, and what is still marked "uncertain".
Why It Works
The weakness. Models struggle to judge code head-on. They have to guess how it behaves on every input and assess security in one go. And if you also ask the model to write tests with expected results, it often locks in what the code does, not what it should do. The test passes, the vulnerability stays. In the experiment, exactly these signals failed to improve real security.
The strength. Models are good at dreaming up malicious inputs and good at reading concrete output: "this request returned the contents of /etc/passwd — that's a leak." Both are easier than "prove this code is secure."
How the method exploits this. It slices the job into two narrow ones. First: invent an attack without a verdict. Second: assess one real run. Actually running the program in between gives you facts instead of guesses. The faithfulness check closes a loophole: the attacker could otherwise fake the vulnerability itself.
Control knobs: - Number of probes (5 in the example) → more probes = wider coverage, more tokens. - Number of rounds (5) → 2–3 is enough for a small change. - The "no evidence ≠ secure" rule → don't remove it: it stops the agent from declaring victory when a probe never reached the code. - List of attack classes (path traversal, foreign id…) → swap in your own: SQL injection, access permissions, secrets in logs. - Judge "doesn't look at the code" → let it look, and the eyeball verdicts come right back.
Prompt Template
The publicly available part of the paper contains no prompts (they are in the appendices). The template below is built from the description of the FALCON architecture.
You are reviewing code you wrote for this task: {task}.
Work in separate modes.
<Attacker>
Target: {function/class/endpoint and file}.
Write {num_probes} deterministic scripts in {probe_folder}.
Each one calls ONLY the target with a malicious or edge-case input:
{attack_classes}.
Rules:
- Do NOT write an expected result and do NOT give a verdict.
- The script only prints: input, return value, exceptions, side effects.
- Do NOT replace or redefine the target or any modules on the path to it.
- Do NOT perform the sensitive operation yourself in the script body.
</Attacker>
<FaithfulnessCheck>
Before running, discard scripts that replace the target, simulate
the violation themselves, or don't call the target. Rewrite them.
</FaithfulnessCheck>
<Verifier>
Run the scripts. Do not read the target's source code.
Judge only by the output. For each probe:
insecure (quote from the output as evidence) / secure / uncertain.
If the target was never called, that is "no evidence", not "secure".
</Verifier>
<Loop>
For each insecure, write up a counterexample: script, output, reason.
Fix the code based on the counterexamples. Repeat. Maximum {num_rounds} rounds.
Show a summary: what was found, what was fixed, what remains uncertain.
Don't break existing functionality: after the fixes, run {test_command}.
</Loop>
What to fill in: {task} — what the code does. {num_probes} — usually 3–5. {attack_classes} — the list of threats for your code. {num_rounds} — 2–5. {test_command} — how to run your regular tests.
🚀 Quick start — paste into a chat or an agent:
Here is a template for checking code security through probes and output analysis (the FALCON method).
Adapt it to my task: [your task and code].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the code is and what language it's in, which threats matter, and how to run the code and the tests. That way the probes target real risks and the runs are actually possible. It will take the rest of the structure from the template.
Limitations
⚠️ Code execution required: "judge by the trace" works only if the agent actually executes the probes. In a plain chat without tools, the model will imagine the output — and the same unreliability comes back. Workaround: run the scripts yourself and paste the output to the judge.
⚠️ Full FALCON is engineering: per-vulnerability detectors, the sandbox, automated faithfulness checks and reinforcement learning are all code. The template above is a simplified version, and the paper did not measure its effect. The gains reported in the paper belong to the full system.
⚠️ Smaller effect in real agents: on standalone functions, security improved several-fold. In full agents working on entire repositories (Claude Code and others), the gain was in the single-digit percentage points. Functionality was largely preserved.
⚠️ "Didn't find anything" ≠ "secure": no violation in the trace only means no observed violation. It is no substitute for an audit and professional penetration testing.
⚠️ A self-judge goes blind: in the study, the attacker and the judge were the same model. They share blind spots: whatever it never thinks to attack, it will never find.
How It Was Studied
The idea was to test the security signal itself, not just the final code. The authors took four ways of evaluating code: FALCON, an LLM judge reading the source, an "attacker" with expected results, and the CodeQL static analyzer. The same coder (GPT-5.4-mini) spent five rounds fixing code against each signal. The real security and functionality tests were hidden from all of them. Every signal reported progress, but the hidden tests told a different story. FALCON gained +25 points (from 61 to 87). CodeQL — about +7. The LLM judge on code — zero. The attacker with expected results — minus 1.7, even though its own metric went up. The agent learned to please the signal rather than fix the vulnerabilities.
Then they tested scale. On two benchmarks, FALCON raised secure-and-functional solutions by 22–25 points for models of different sizes. The small GPT-5.4-nano caught up to top models without repair, and in places surpassed them. In three real agent harnesses (SWE-agent, OpenCode, Claude Code) on 186 tasks, the gain was smaller: 1–5 points. The verifier was tested separately: 88% accuracy on execution traces, 61% on source code, and the source-code judge was also more expensive. Finally, FALCON was used as a reward for training small models. There too it was less prone to over-optimizing the signal. That part is not applicable for readers.
The surprise: the LLM judge and the attacker with self-written tests didn't help at all, and sometimes hurt. Practical takeaway: don't trust an agent's self-assessment that "the code is secure", or tests it wrote for itself with expected results.
Adaptations and Extrapolations
🔧 Technique: a human runs it, the model judges → works in a chat without an agent
Ask the model to write probe scripts following the Attacker rules. Run them yourself and paste the output into a new request: "Below is the run output. Don't look at the source code. Give a verdict for each probe with a quote from the output." This way the judge evaluates real facts, not imagined ones.
🔧 Technique: add a fourth verdict → fewer false wins
Keep the rule in
<Verifier>: "if the target wasn't called → no-evidence". In the paper this is the key safeguard: the agent can't "win" just because every probe got rejected.
Extrapolation (beyond the original, untested): the same principle — "separate finding problems from the verdict, and give the judge facts" — can be applied to other agent checks, for example edits to data migration scripts:
First generate 5 test scenarios for the migration {description}: only the input and the run command, no expected result.
Run them on a copy of the data. Then, without looking at the migration code, evaluate only the output:
ok / violation (with a quote) / no data.
Resources
- Paper: Secure Agentic Coding through Counterexample-Grounded Feedback (FALCON)
- Authors: Tue Le, Anh Nguyen Hoang, Gabriel Sasseville, Doseok Jang, Pierre-Luc Bacon, Nghi D. Q. Bui
- Organizations: QACI (FPT Corporation), Mila — Quebec AI Institute, Université de Montréal, Google
- Benchmarks from the paper: CWEval, SecCodeBench-V2, SusVibes
- Tools mentioned: CodeQL, SWE-agent, Claude Code, OpenCode