Practical prompting, backed by research

The paradox: instruction files like AGENTS.md don't actually make code safer. A large-scale study found they only inflate token usage by 20% — and sometimes even hurt the output. Specification Frame solves this differently: it lets you get code from a model without the classic bugs — money handled as float, dates missing timezones, payments charged twice — using one fixed 267-word block. The block doesn't describe HOW to write the code, but WHAT must be true in the result. Tested on 5 models from 5 different companies — errors dropped across all five, with zero regressions.

AI writes code that works, but it doesn't think about security on its own — passwords in plain text, secret keys hardcoded right into files, a JWT token signed with a hardcoded password, and anyone can forge someone else's session. in this 6-app, single-run study the security-aware versions had no critical/high findings, with no edits — just by adding a ready-made requirements block to the end of your prompt. The trick is that the model follows specific instructions extremely well — it just doesn't consider security part of the task unless you name it explicitly. Simply writing "password hashes," "secrets from environment variables," "parameterized queries" — and two critical and eleven serious vulnerabilities disappeared from the output.

You ask the LLM to 'write tests for this code' — and you get tests that would pass even on broken code. The model copies the code's logic instead of checking what it's supposed to do. Spec-Driven Test Generation lets you find missing edge cases through an explicit behavior contract — before you write a single test. The model spells out preconditions and postconditions, marks what's already covered (checkmark) and what isn't (X), and writes tests only for the gaps — it can't write a test 'just to have one,' only for a specific uncovered condition.

Three out of sixteen code samples that Claude wrote without a single mention of security turned out to be real vulnerabilities — an exposed debugger with remote code execution and endpoints leaking every user's personal data with no permission checks. One phrase in the prompt about the SOC 2 standard (or GDPR, HIPAA) gets you code that closes these holes on its own — no follow-up requests or fixes needed. The model solves the task literally — if you didn't say the data was sensitive, it won't treat encryption and logging as part of the job, even though it perfectly knows how to do them. Just naming the standard — compliance jumps from 47-88% to 86-100%.

Self-Critique delivers exactly 0% improvement on half of the unfinished projects. That's not a bug; it's the method's boundary. The framework lets you pick a code-improvement strategy that works instead of trial and error. The decision framework is a 'failure type → method' map: logic gaps (a forgotten function) → Self-Critique (the model spots them in review); external integration (RabbitMQ not configured) → RAG-Assisted (drop a tutorial into the context). The evidence: Self-Critique 0% vs. RAG 100% on integration, across 18 experiments on real GitHub projects.

Add 5 code style requirements (line length, docstrings, pathlib instead of os.path) and functionality drops by 6%, even though the requirements have NOTHING to do with whether the code works. Even top models satisfy all 5 instructions at once only 47% of the time. VeriCode, a taxonomy of 30 automatically checkable instructions (style, logic, documentation, error handling), lets you measure not just whether code works but whether it follows the requirements. Each instruction is checked via linters and AST analysis and returns pass or fail. The researchers extended the classic BigCodeBench and LiveCodeBench benchmarks with these instructions, creating Vibe Checker — an environment that evaluates both dimensions.

The discovery: Even top LLMs are terrible at processing JSON directly – GPT-4o maxes out at 77% accuracy. Models get lost in the structure, confuse similar keys (name, room_name, full_name), and stumble over nesting. As the size grows to 80K tokens, Mistral-large drops by 91%. The method lets you extract data from complex API JSON responses with up to 95% accuracy – especially for filtering and aggregation. The trick: instead of "read the JSON and give me the answer" → "write a Python function that parses the JSON". The code runs – the result is exact. Add the JSON schema to the prompt – +12% accuracy. Even a trimmed structural sample works if the full response doesn't fit in the context.

5 lines deleted, 60 added, all for a one-line fix: `<` to `<=`. GPT-5.4 saw a bug in a 10-line function and bolted on validation, type casting, and a curve recalculation while it was at it. All the tests passed, but that's not a patch anymore; it's an unrequested refactor, which the researchers call over-editing. The method lets you keep the model within the scope of the task: one addition to the prompt, "preserve as much original code as possible, change only what's necessary", cuts the volume of unnecessary edits by almost a third.

GPT-4o drops from 97% to 10% accuracy when a problem has millions of possible combinations. Logic puzzles, calculations with dozens of interdependent parameters, problems that require trying out options — the model starts hallucinating and contradicting itself. The reason: an LLM generates text token by token and can't backtrack. If it picks the wrong path, it keeps generating anyway, inventing facts to plug the logical holes. Symbolic solvers move the search for a solution out of text generation and into code: the model only translates the problem's conditions into Python or Prolog, and a library (SymPy for equations, Constraint for logic) systematically works through the options, backing up at dead ends. Accuracy on hard puzzles rises from 10% to 89%.

Identical examples, one request — but accuracy jumps from 0% to 100% depending on where the examples sit in the prompt. The co-located tests method lets you get correct code on the first try — no iterations, no fixes. The trick: examples inside the docstring sit a line away from the function — the transformer processes them as a single whole. Move the tests to the end of the prompt and the link between the example and the task weakens by 2.8–4.4 times, and the model starts guessing the behavior instead of following the examples.