TL;DR
The researchers took the security settings of real coding agents (Claude Code, Codex CLI, Gemini CLI and others) and switched each one on and off in turn. They ran the same task in a "poisoned" environment: malicious instructions in repository files, in tool descriptions, in external services. This shows exactly what cuts down attacks and what it costs in useful work. A harness (the agent's scaffolding) is the program around the model: it assembles the context, hands out tools and decides what can run without asking.
The headline finding: auto-approving actions makes the agent noticeably more "productive" and at the same time almost completely defenseless. With it, the attack success rate rose from 29.2% to 95.6%. The agent doesn't just "occasionally slip up": any malicious instruction from the environment (from an issue, a README, a service response) almost always gets executed. The reason is simple: the model reads repository text as part of the task, and human confirmation is the last barrier, which you just removed.
The other settings behave differently. Network isolation and read-only mode nearly zero out attacks but hit useful work hard. A command allowlist protects at a small cost in usefulness. A denylist of dangerous commands protects and even improves the result. There's also a trap: you forbid one operation, and the agent does the same thing another way.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Pick a security setting → turn it on / off (one at a time)
STEP 2: Run the same real work tasks in both modes
STEP 3: Measure separately: (a) did the agent do the required work,
(b) did the malicious instruction from the environment fire
STEP 4: Compare the pairs → protection vs. cost in usefulness
STEP 5: Check workarounds to the forbidden operation
For you, this is the manual version: don't pick settings by gut feel; compare protection against cost on 3–5 of your own typical tasks.
Example in Practice
Task: A small agency in Austin hooked Claude Code up to an online store's repository. Contractors and random people post in the public issue tracker. The developer is sick of the endless "Allow?" prompts and wants to turn on "just do everything yourself" mode. You need to decide which settings to keep.
Prompt:
You are a security auditor of coding agent settings.
Context: we use Claude Code on an online store's repository
(Node.js, PostgreSQL, deploys via GitHub Actions). Outside people post
in the issue tracker; the agent reads issues and the README. The developer
now wants to turn on auto-approve so the agent doesn't ask permission for every command.
For each setting below:
1. Explain which attack it protects against in our situation.
2. Estimate the cost: what legitimate work it will break or slow down.
3. Name 2-3 workarounds to the forbidden operation
that this setting does NOT close.
4. Give a verdict: enable / enable partially / don't enable.
Settings: auto-approve, network isolation, read-only mode,
command allowlist, command denylist (rm -rf, curl | sh),
path restrictions, MCP server permissions.
At the end, assemble the final configuration and list
what I should verify by hand after setting it up.
Result: The model returns a table for each setting: protection, cost, workarounds, verdict. At the end, a recommended configuration. Most likely auto-approve will be flagged as dangerous without isolation, and the "verify by hand" list will include test requests for forbidden actions.
Why It Works
The weakness. The model can't tell the owner's instruction from an instruction hidden in an issue's text or a service response. To the model, it's all just text in the context. So if you remove confirmations, the malicious instruction gets executed. For the same reason, "just forbid it" often doesn't work: the action the agent wants can be carried out by another route.
The strength. The harness, not the model, decides what actually runs. Its rules (permissions, sandbox, command lists) hard-limit actions regardless of what the model "wanted." These rules are measurable: turn it on → measure → turn it off → compare.
How the method uses this. By comparing on and off modes on the same task, you see the cost and benefit of each setting separately. That lets you choose targeted restrictions (command allowlists and denylists) over blunt ones (full isolation) that protect but break the work.
Knobs to turn: - Auto-approve → keep it only inside a sandbox with no network access. - Network isolation → turn it on if the agent reads untrusted content and shouldn't download anything. - Read-only mode → for code analysis and review, not for edits. - Command allowlist → when tasks are predictable and the command set is known. - Command denylist → cheap insurance on top of everything else.
Prompt Template
The paper has no ready-made prompts for readers: its output is data and settings. Below is a template I put together from its conclusions. It's meant to walk you through the audit method, not to represent a technique from the authors.
You are an auditor of a coding agent's security settings.
Agent: {agent_name}
What it does for us: {typical_tasks}
What the agent reads from untrusted sources: {issues, README, web, MCP, packages}
What we must protect: {secrets, production, database, repository}
What we plan to enable: {planned_setting}
For each setting (auto-approve, network isolation, read-only,
command allowlist, command denylist, path restrictions, MCP permissions):
1. Protection: which attacks from the "reads from untrusted sources" list it stops.
2. Cost: which of {typical_tasks} it will break or slow down.
3. Workarounds: 2-3 alternative ways to do the same forbidden action.
4. Verdict: enable / partially / don't enable, and why.
Then:
- Assemble the final configuration with specific {agent_name} flags/keys.
- Write 5 test requests I can use to check that the restrictions actually work
(including workarounds).
- Flag settings you're not sure about: I'll check them against the documentation.
What to fill in: the tool name, 3–5 typical tasks, sources of untrusted text, what you can't afford to lose, what you want to enable.
Limitations
⚠️ One model: All 2,500 runs used a single base model (GLM-5.2) to isolate the harness effect. The paper doesn't show how the same settings behave with other models.
⚠️ A modest task set: Only 23 tasks. A real project is more complex, so measure the cost in usefulness on your own repository.
⚠️ Settings, not prompts: The paper is about harness flags and configs. It didn't test what text in CLAUDE.md or a system prompt does against attacks. The conclusion "write 'don't listen to external text' in the instructions" does not follow from it.
⚠️ Closed products are a "black box": For paid agents, more than half the cells are "unknown." For the injection filter, it's 13 out of 13 closed cells. Don't assume protection exists if it isn't confirmed.
⚠️ Goes stale fast: Settings and their defaults change with every version. The available text of the paper lacks the sections with attack details and per-agent failure analysis, only the summary numbers.
How They Studied It
The team first mapped out the defenses. They studied 40 coding agents (27 open-source and 13 closed) and identified ten mechanisms: auto-approve, network isolation, command allowlists and denylists, MCP permissions, an injection filter, path restrictions, read-only mode, an audit log and project trust. Then they assessed 400 "agent × mechanism" cells: on by default, must be enabled manually, absent, unknown. The assessment was done independently by three LLMs and two humans, with a mandatory link to a line of documentation or code. This produced a curious picture. About half of the confirmed defenses have to be enabled manually: read-only mode is opt-in almost everywhere (30 of 31), and so is auto-approve (26 of 30). MCP permissions and auditing, on the other hand, are usually already on. Closed products have "unknown" in more than half their cells, versus 16% for open ones.
Next they built a benchmark of 23 tasks (Python, JS, C++, C#, Go, Ruby). Malicious resources were mixed into the environment via five routes: repository files, instructions and skills, tool descriptions, memory and knowledge, external services. The user's legitimate instructions were left untouched. Separate deterministic checks measured two outcomes: was the work done, and did the attack fire. Each setting was run on and off in identical Docker containers. In total: 2,500 runs, 81,155 tool calls and 2.2 billion tokens.
Results: - Auto-approve: attacks 29.2% → 95.6%, usefulness went up. - Network isolation: attacks 57.1% → 0.8%, but a 24.5 pp loss in usefulness. - Read-only mode: attacks 8.3% → 0.8%, a 34.0 pp loss. - Allowlist: a small loss in usefulness. - Denylist: usefulness actually went up.
The unexpected and practical part: restricting a "general capability" cuts both the harmful and the needed, because both operations go through the same channel. And an allowed tool leaves a loophole to a forbidden one via a workaround. The practical takeaway: evaluate each setting on three axes at once: attacks, usefulness, execution cost.
Adaptations and Extrapolations
🔧 Technique: add "workarounds" → check for loopholes
After the verdict on each setting, ask:
For each forbidden operation, come up with a way to perform it with another
tool or command. Show me how to verify that the restriction closes it.
The paper directly shows that allowed tools leave forbidden actions reachable. This step looks for those holes before an attack does.
🔧 Technique: two modes by task type → smaller losses
Ask it to split tasks into "analysis and review" (read-only + network off) and "edits" (sandbox + allowlist). Read-only mode is costly for edits but costs almost nothing for code analysis.
Resources
- HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses? (preprint)
- Project site: https://tsingpig.github.io/HarnessSecurity-Benchmark/
- Authors: Zhengyang Zhu, Liming Huang, Runmin Ji, Mingxi Ye, Zihan Zhou, Hanyang Guo, Jingwen Wu, Yuhan Ye, Yuming Feng, Hong-Ning Dai, Zibin Zheng
- Universities: Sun Yat-sen University, Peng Cheng Laboratory, East China Normal University, Hong Kong Baptist University, Tsinghua University