SWE-Clar: clarify requirements before coding, and test understanding with separate questions

STEP 1: The agent studies the materials and asks the client questions → rounds of "question — verified answer"
        (especially: how the software is run, what the inputs and outputs are)
STEP 2: The client answers, confirms, or corrects the agent's interpretations → the answers
        are checked by a supervisor (accuracy, completeness, didn't go beyond the question)
STEP 3: The agent decides on its own that clarification is done → the context is "frozen"
        (request + dialogue + what the agent saw in the working folder)
STEP 4a: Implementation branch: the agent writes code with no access to the client → acceptance tests
STEP 4b: Understanding-check branch (a separate run): the agent answers questions
        about the requirements → a judge compares them to the reference (correct / partial / wrong)

Steps 1–3 happen in one conversation. Branches 4a and 4b are separate runs from the same frozen context. The check questions must not hint to the agent what to put in the code.

Task: The owner of a cosmetics store on Amazon asks Claude Code for a script that calculates unit economics for every SKU from exported sales reports. The request goes like this: "I need to calculate margin per product so I know what to cut from the catalog." The requirements the client didn't mention are endless: fees, fulfillment, returns, taxes, output format.

Prompt:

You are a freelance developer. The client is me, an Amazon store owner.
My original request: "I need a script that calculates margin per product from my Amazon
Seller Central report exports so I know what to cut from the catalog." The export files are in the /data folder.

PHASE 1. CLARIFICATION (no writing code).
- Study the files in /data.
- Ask me questions in batches of 3–5 until you understand what's needed.
- Be sure to clarify: how I'll run the script, what I feed in, what I want to get out.
- Offer your own interpretation ("Here's how I understood it: ... Right?"), not just open questions.
- Ask about exceptions and edge cases: returns, cancellations, products with no sales.
- When you're ready, write "CLARIFICATION COMPLETE" and list 5–7 requirements in your own words.

PHASE 2. IMPLEMENTATION — only after my "ok." From then on I'm unavailable, no more questions.

Result: The agent won't start coding right away. Instead it sends a series of questions with candidate interpretations: how to calculate fees, what to do with returns, what format the report should take. After your answers, it gives you a list of requirements in its own words. You'll see where it misunderstood, correct it, and only then give the go-ahead for code.

The weakness: given an incomplete task, the model fills in the gaps with guesses and confidently writes code. Without questions, it fails almost every task. But even after the client answers, understanding isn't guaranteed: the model later restates some fully disclosed requirements incorrectly.

The strength: the model is good at writing clarifying questions and restating requirements in its own words. You can check that restatement against what you meant. An error in the restatement predicts a code failure on that requirement.

How the method uses this: - The question phase gathers what's missing while you're available. - A separate restatement of requirements is a cheap check before the expensive implementation. - Isolating the check from the code keeps the control questions from tipping the model off and skewing the check.

Control knobs: - Interpretations instead of open questions ("Here's how I understood it...") → the client confirms or corrects faster. - Mandatory topics (how it runs, input, output) → plug the most expensive holes. - Question batch size → smaller batches mean more rounds, but less confusion. - Answer review (someone else checks your answers for accuracy and completeness) → fewer errors in the brief itself.

The available text of the paper has no full prompts (the roles are in Appendix F). The template below is built from the mechanics the paper describes: mandatory clarification of critical details, confirmation of interpretations, no code until clarification is over.

You are a {contractor_role}. The client is {who_the_client_is}.
Original request: "{initial_request}". The materials are in {folder_or_files}.

PHASE 1. CLARIFICATION (no writing code).
- Study the materials and figure out what the request is missing.
- Ask questions in batches of {number}.
- Be sure to clarify: how the result is run, what goes in,
  what should come out, how to handle {critical_exceptions}.
- Offer your own interpretation ("Here's how I understood it: ... Right?").
- When you think you have enough, write "CLARIFICATION COMPLETE" and list
  the requirements in your own words.

PHASE 2. IMPLEMENTATION — only after my "ok."
The client is unavailable; there will be no more questions. Build {deliverable} and test it yourself.

Placeholders: - {contractor_role} — developer, analyst, writer. - {initial_request} — your real, incomplete request, as is. - {critical_exceptions} — the "sharp corners" you already know about: returns, cancellations, empty data.

🚀 Quick start — paste into a chat:

Here's a template for a clarification phase before implementation. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what the task is, what materials you have, how the result will be run, and which exceptions you already know about. That way the clarification phase goes straight for the most expensive holes instead of generic stuff.

⚠️ It's a benchmark, not a proven technique: the authors measured how models cope; they didn't test whether "restating requirements before coding" specifically improves results. The link between a correct answer and working code is shown as a correlation.

⚠️ The client in the experiment is a model: the client, the client's reviewer, and the judge were all played by the same LLM. A real person answers worse, less consistently, and more slowly.

⚠️ Clarification doesn't close the whole gap: it recovers only part of the lost performance. A full spec up front is still noticeably better. Questions don't replace a good brief.

⚠️ Answers and code are different things: sometimes a requirement is stated correctly but the code doesn't implement it. And the reverse: the answer is incomplete but the code passes the test. A restatement is a signal, not a guarantee.

⚠️ Narrow scope: new projects from scratch, local development, the client available only at the start. For changes to existing code and ongoing work with a client, the conclusions transfer less well.

The Upwork team took 100 real greenfield jobs worth $4,000 and up. From records of client–freelancer conversations, a model extracted the requirements critical to delivering the work: both stated outright and inferred from context. From those they built 512 control questions with reference answers, plus acceptance tests. Nine top models acted as freelancer agents. A single model played the client, supervisor, and judge. The idea was to separate understanding from execution: after the dialogue the context was frozen and two isolated branches were launched.

The most telling part is the comparison of three conditions on two weaker models. With no questions to the client, the agents passed 5–6% of tasks. With the clarification protocol, 39–49%. With a full spec up front, 71–75%. Clarification closes 51–62% of the gap, and the rest is the price of details nobody mentioned. The best model passed 67% of tasks, the worst 39%. The higher the score for restating requirements, the better the code result: a 0.89 correlation across nine models.

Next they compared answers and code requirement by requirement. The test pass rate for a requirement with a correct answer was 86–91%. With a wrong answer, 34–68%, noticeably lower. But the link isn't perfect: 6–8% of requirements with a correct answer still fail in code. Something else was surprising: for 3–6% of requirements the client fully disclosed, the agent still states them wrong. Disclosing information and understanding it are not the same thing.

🔧 Technique: add a control restatement → catch misunderstandings before code

In the paper, the understanding check is a separate run. A practitioner can do the same in everyday work. After "CLARIFICATION COMPLETE," ask:

Answer three questions in writing, without looking at the code: 
1) What does the program do in the case of {edge_case_1}? 
2) What does it do if {edge_case_2}? 
3) How is it run and what does it return? 
Point out where you're not sure.

You compare the answers with your expectations and correct them before implementation. This is an extrapolation: the paper showed that wrong answers predict failure, but didn't test this technique as a way to prevent it.

🔧 Technique: a second chat as a "supervisor" → fewer errors in your answers

In the paper, the client's answers are checked by a separate agent: accuracy, completeness, didn't go beyond the question. If you're answering a lot of the agent's questions, send your answers to another chat and ask it to find contradictions and gaps. Then send the corrected version back.

• SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software? (Preprint, under review)
• Code and data: https://github.com/upwork/SWE-Clar
• Authors: Jialei Li, Dingxian Wang, Xiaoyan Zhao et al. — University of Science and Technology of China, National University of Singapore, Upwork Inc.
• Related benchmarks mentioned in the paper: SWE-Lancer, ClarifyCodeBench, Ambig-SWE, SWE-Together, SWE-Interact, ICAE-Bench