Harness-Context Study: structured context in an agent's harness gives weak models a big boost and top models almost nothing

STEP 1 (Scan):       quick triage of suspicious files (regexes/heuristics) → list of candidate files
STEP 2 (Process):    agent reviews files in a batch, receives a context block → findings with severity and explanation
STEP 3 (Revalidate): a separate session re-checks each finding → labels: true-positive / false-positive / fixed / uncertain
STEP 4 (Merge):      run with different models, 2 times each → merge duplicates, different descriptions of one bug collapse into one

Steps 2 and 3 run as separate sessions (different requests). The context block is assembled without an LLM, deterministically, through static code analysis.

Task: A marketplace team on Node.js is preparing to release its payments module: order intake, promo-code discount calculation, a call to the payment gateway. The pentest is two months out, but the code needs checking now. In Cursor or Claude Code they have a cheap model and a strong one.

Prompt (Process stage, cheap model, with context):

You are a security researcher. Review the file src/payments/applyPromo.ts
for vulnerabilities an external attacker could exploit.

<Code_context>
Entry points: POST /api/orders, POST /api/promo/apply
Who calls applyPromo: orderController.create, cartController.recalc
What applyPromo calls: db.promoCodes.find, pricing.calcDiscount, gateway.charge
User input sources: req.body.promoCode, req.body.items[].qty, the X-Forwarded-For header
Dangerous spots (sinks): SQL query in promoCodes.find, gateway.charge(amount)
Checks along the path: authMiddleware (present), rateLimit (only on /api/orders)
</Code_context>

For each finding, give: severity (Critical/High/Medium/Low),
file and lines, an explanation, how an attacker would reach that spot.
If it's just a logic bug with no path for an attacker, mark it as Bug, not as a vulnerability.

Prompt (Revalidate stage, new session, same or different model):

Here is a finding from the previous analysis: {finding}.
Open the code and re-check it. Is there a protection the analysis missed?
Answer with one label: true-positive / false-positive / fixed / uncertain,
and explain why in 2-3 sentences.

Result: After Process, the model gives you a list of findings with severity, code lines and a description of the attacker's path. Some of it will be noise. After Revalidate in a separate session, each finding gets a label with a short justification. Repeat on the strong model and merge duplicates: models will describe the same hole differently. The cheap model with a code map yields more findings, the strong one fewer false ones. Models usually inflate severity, so you'll need to review it yourself.

The weakness: a model handed a file "cold" sees only that file. A vulnerability often lives between files: input arrives in one place, the dangerous call happens in another. A weak model doesn't connect those dots, so findings end up either shallow or made up.

The strength: if the connections are already written out as text (who calls what, where input comes from, where the protections are), the model reasons over them very well. All it has to do is judge whether there's a path from input to the dangerous spot. That's easier than reconstructing the project structure itself. A top model reconstructs it on its own, so the map adds almost nothing for it. The information is duplicated, and the gain is within noise.

How the method gets around the weakness: the context takes over the "find the connections" job that a weak model can't handle. A separate verification session filters out noise. Running several models covers each one's blind spots.

Your levers: - Context size → don't dump everything in. Targeted context (calls, input, dangerous spots) beats a "pile" of docs and configs. This is a general pattern from related work: excess context hurts. - The model → give a weak one the code map; a strong one just needs a good prompt. That saves prep time. - Number of runs → run at least twice: one run reproduces only most of a model's findings. - Model lineup → add a cheap model instead of a second expensive one. Coverage goes up, cost barely does. - Re-check labels → add your own (for example, "needs a human") so questionable items don't get lost.

This is a reconstruction based on the paper's description of the stages, not the verbatim original: the full prompt texts aren't given in the accessible part of the study.

<Role>
You are a security researcher. You look for vulnerabilities an external attacker could exploit.
</Role>

<Task>
Review these files: {file_list}
</Task>

<Code_context>
Entry points: {routes, handlers, middleware}
Who calls the code in these files: {callers}
What these files call: {callees}
Attacker input sources: {request fields, headers, files}
Dangerous spots (sinks): {SQL, shell, file system, payment calls, eval}
Checks along the path: {authorization, validation, rate limits}
</Code_context>

<ReasoningLogic>
For each dangerous spot:
  trace the path from the input source to it;
  check whether there's a protection along the path;
  if there's no path for an attacker, it's not a vulnerability.
Mark a logic bug with no attacker path as Bug, not as a vulnerability.
</ReasoningLogic>

<Output>
For each finding: severity (Critical/High/Medium/Low/Bug), file and lines,
explanation, attacker path.
</Output>

Re-check stage (new session):

Here is a finding: {finding}.
Open the code and re-check it. Is there a protection the analysis missed?
Answer with a label: true-positive / false-positive / fixed / uncertain, and explain in 2-3 sentences.

What to plug in: take {file_list} from the quick triage (grep for dangerous calls: exec, query, eval). Build the <Code_context> block from the real code.

🚀 Quick start — paste into a chat (ideally Claude Code or Cursor with access to the project):

Here's a template for vulnerability hunting with code context. Adapt it to my project: {describe the project and stack}.
First, build the <Code_context> block from the repository: entry points, who calls whom,
input sources, dangerous spots, checks along the path. Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your stack, entry points and which files are critical. Those questions feed the code map the whole method rests on. It'll take the pattern from the template and build the context for your project.

⚠️ One repository: the main experiment ran on a single codebase (TypeScript, an API gateway for LLMs). Part of the effect was checked on another task set, but there's no broad confirmation.

⚠️ A mixed comparison: the "minimal prompt → engineered prompt with map" comparison changes both the prompt and the context at once. The authors isolated the map's pure contribution with a separate comparison, but the 20–86% figures cover both changes together.

⚠️ The map was built by a static analyzer, not an LLM: in the paper, CodeQL assembled the context. If you ask the agent to draw the map itself, that's untested: it may be incomplete or wrong.

⚠️ Severity is inflated: models most often assign findings a higher severity than maintainers later accept. Have a human look at every finding, especially Critical ones.

⚠️ Self-verification isn't perfect: re-checking your own findings is a filter, not proof. The model can discard a real vulnerability and let a false one through. For external reports you need confirmation by actually running an exploit.

⚠️ No universal recipe: the results are about specific models at the time of the study. Which models count as "weak" and "strong" changes fast. The principle (check whether context helps your model) is more durable than the numbers.

⚠️ Expertise required: for security and code review without a human reading the findings, the method doesn't replace a specialist.

The authors took the open-source DeepSec scanner from Vercel Labs and ran it on real production code: an API gateway for LLMs, roughly 150 candidate TypeScript files. Its vulnerabilities were not public at scan time, so the model couldn't just recall a known CVE from its training data. Benchmarks usually use already-patched bugs, where success may be memory rather than discovery.

First they tested 24 model + run combinations. All findings were merged into a canonical list: 265 unique vulnerability root causes and 511 logic bugs. No single combination covered everything. The best two runs together yield only 60.4% of the list, and 14.3% of findings were discovered by only one model.

Then they picked four models ranging from top-tier to small (GPT-5.5, Composer-2.5, GLM-5.1, Gemma-4-31B) and compared eight conditions: two baseline prompts and six context variants. Only the Process stage changed; everything else was held fixed. The gain shrank as model strength grew: +86% for the small model, +36% for the mid-size, +20% for the cheap strong one, +2% for the top one. That last result falls within run-to-run noise. The direction replicated on a held-out sample from CyberGym.

The surprise was that the top model didn't find more vulnerabilities; only its false positives went down. In practice that means paying for the top tier makes sense for a cleaner report, not for wider coverage. One more takeaway: a single run recovers only 68–91% of a model's two-run set, meaning one run is unreliable.

💡 Adaptation for code review in Cursor/Claude Code: the same "context for the weak model, cleanliness from the strong one" principle works beyond vulnerabilities. If you're reviewing a pull request with a cheap model, first give it a list of the affected functions, their callers and the tests. A strong model just needs the diff. The paper didn't test this — it's the principle carried over to a different task.

You're reviewing a pull request. Here's the diff: {diff}.
Context: functions that call the changed code: {callers}.
Tests covering these spots: {tests}.
Find only the problems that break behavior for the callers. If you're not sure, mark it uncertain.

🔧 Technique: different models at different stages → cheaper at the same coverage. A cheap strong model does the broad pass (Process), the top model only re-checks the findings (Revalidate). In the paper each model verified its "own" findings; mixing models across stages is our extrapolation.

Extrapolation: ensemble + re-check. Run the same task with two or three models. Merge the findings with one request:

Copy

Here are findings from three models: {list}. Merge duplicates: treat one problem described in different words as one.
For each unique problem, state how many models found it.

Don't throw out problems found by only one model — check them by hand: in the paper, exactly those make up a noticeable share.

• Title: How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery
• Authors: Ganni Galea Curmi (Georgian; University of Toronto), Ben Wilde (Georgian)
• Venue: Proceedings of Machine Learning Research 349, 2026, Conference on Applied Machine Learning for Information Security
• Tools from the paper: DeepSec (Ubl, 2026, Vercel Labs), Trailmark (Arciszewski, 2026), CodeQL, the CyberGym benchmark (Level-0), ZeroDayBench
• Related work: Big Sleep (Google Project Zero), DARPA AI Cyber Challenge