Compiler-Guided Refinement: Precise Feedback Beats General Instructions

STEP 1: The model generates a solution
STEP 2: You run it through a verifier → get the exact error
STEP 3: You copy the error in full → add it to the prompt → ask for a fix
STEP 4: The model generates a corrected version
STEP 5: You repeat steps 2-4 until it passes the check

Each round is a separate request to the model with the accumulated history.

Task: You're writing an SEO-optimized product listing for Amazon. It has to fit the platform's limits and pass review: title up to 60 characters, description of 500-2000 characters, no banned words ("guarantee", "100%", "best"), and the product specs must be included.

Prompt, round 1:

Write a product listing for Amazon:

Product: TWS wireless earbuds with noise cancelling
Price: $29.99
Features: 
- Active noise cancelling (ANC)
- 24-hour battery life with the case
- Bluetooth 5.3
- IPX4 water resistance

Amazon requirements:
- Title up to 60 characters
- Description 500-2000 characters
- Not allowed: "guarantee", "best", "100%", emoji
- Required: specs in the text

→ The model produced 320 characters (too short) and used the word "best"

Prompt, round 2:

I checked the text and found problems:

1. The description is 320 characters — it needs at least 500
2. The banned word "best" appears in the sentence "The best choice for workouts"
3. IPX4 water resistance isn't stated explicitly

Fix these 3 problems. Leave everything else as is.

→ The model fixed them, but now it's 2100 characters (over the limit)

Prompt, round 3:

The description came out at 2100 characters — the max is 2000.
Cut 100 characters without losing any specs.

Result: After 3-4 iterations you'll have a listing that passes every Amazon check. Each round: a specific error → a specific fix.

LLMs are bad at juggling lots of constraints at once. Give a prompt like "write 500-2000 characters without words X, Y, Z and with features A, B, C" and the model will lock onto 2-3 things and drop the rest. That's not a bug, it's how generation works: the model optimizes locally (the next word), not globally (all the rules at once).

A compiler (or any verifier) is external memory that remembers ALL the rules and checks EVERYTHING. The model tries → the verifier says exactly what's wrong → the model focuses on that one problem → fixes it. On the next iteration the verifier finds the next problem — and so on until everything checks out.

Why a precise error beats documentation: "undefined function foo on line 12" gives context (where, what, what it's called). "Remember to define functions" gives no clue where to look. Models are good at local edits when the location is pinpointed — they don't need to understand the whole system, just the specific note.

Control levers: - Number of iterations — 2-3 is enough for simple tasks, hard ones can take 10-15. Save tokens on the easy stuff, be patient on the hard stuff. - What to copy from the error — the whole validator output or just the specific line. The more precise the pointer → the faster it converges. - When to stop — you can set a criterion ("until 100% of checks pass") or a cap ("5 attempts max").

Round 1 — Generation:

Do {task} following these requirements:

{requirements_list}

Context:
{task_data}

Rounds 2+ — Fixing:

I checked the result with {verifier}. Found these problems:

{exact_check_output}

Fix these problems. Don't touch anything that already works.

What to fill in: - {task} — what you're making (text, formula, config, table) - {requirements_list} — every constraint you know up front - {verifier} — what does the checking (linter, Hemingway Editor, Excel formula, checklist) - {exact_check_output} — paste it as is, don't paraphrase

⚠️ You need a "strict checker": The principle doesn't work for subjective judgments ("make it more creative", "doesn't sound convincing"). You need a check with precise output — a compiler, a test, a validator, an SEO analyzer, a clear yes/no checklist.

⚠️ Lots of iterations: Hard tasks can take 10-15 rounds. If you need it fast, this method isn't for you. If you need it right, be patient.

⚠️ Cascading errors: In the study, 2 problems didn't get solved even in 20 attempts — fixes in one place broke something else. If you're going in circles after 5-7 iterations, stop and reframe the task.

Researchers at USC took the Exercism platform (programming practice problems) and tested GPT-5 on 56 problems in Idris — a rare, strongly typed functional language. For context, they also checked popular languages: Python (the model solved 90% of problems), Erlang (74%). Idris was a flop: 39% without any help.

Then they compared 4 strategies for helping the model. Method 1: copy the errors from the Exercism platform into the prompt (1 iteration and 5 iterations) — raised it to 48%. Method 2: give the model a guide to "common Idris errors and how to avoid them" via a vector store (RAG) — 54%. Method 3: give it the official language documentation — 61%. Method 4: local compilation + pasting compiler errors into the prompt, up to 20 iterations — 96%.

Why such a gap? The researchers analyzed the error types. The most frequent: "undefined name" (123 times), "ambiguity" (111 times), "syntax" (33 times). These aren't deep logic failures — they're surface problems with imports, function names and module visibility. The docs explain "how the Idris import system works" but don't tell you what's wrong on your line 12. The compiler says "function foo is undefined on line 12" — it's obvious where to push.

A surprising insight: problems needed on average 7-8 iterations to compile and 1-2 iterations after that to pass the tests. In other words, 95% of the effort goes into clearing the "hard barrier" (the compiler), 5% into logic (the tests). That explains why in production tasks (copy, formulas) precise constraints (an SEO validator, a client's checklist) work better than vague ones ("make it good").

🔧 Technique: Midway check-ins → token savings

Instead of 20 blind iterations, you can check in midway:

Let's pause after 3 attempts. Show me:
1. Which problems you've already fixed
2. Which problems are left
3. Why they keep happening

Then we'll continue.

Effect: You see the error pattern → you can nudge the model in the right direction ("look, you keep forgetting the character limit — focus on that").

🔧 Technique: Two-stage feedback

Combine a soft + a hard verifier:

Step 1: The model generates → you read it → you give meaning-level notes ("forgot about the audience", "wrong tone")

Step 2: The model revises → you run it through a hard verifier (Hemingway Editor, checklist) → precise errors → final fix

Effect: First you nail the strategy (what to say), then the tactics (how to say it).

🔧 Technique: The model as the verifier

If there's no ready-made validator, write a checklist and ask the model to check itself:

Check your text against this checklist:
- [ ] Title up to 60 characters
- [ ] Description 500-2000 characters
- [ ] No words: "guarantee", "best", "100%"
- [ ] All specs present: ANC, 24h, BT 5.3, IPX4

For each item write ✅ or ❌ and what needs fixing.

Then:

Here's your check:
[paste the output]

Fix everything marked ❌.

Effect: The model finds the problems itself → fixes them itself. It works worse than an external validator (the model can miss its own mistake), but it beats nothing.

Compiler-Guided Inference-Time Adaptation: Improving GPT-5 Programming Performance in Idris The study uses the Exercism platform for testing and references prior work on MultiPL-E (a multilingual code generation benchmark), Self-Refine (iterative improvement through self-critique), DrRepair and Baldur (using compiler errors to repair code).

Authors: Minda Li, Bhaskar Krishnamachari — Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California.