VeriCode: Verifiable Instructions for Evaluating LLM-Generated Code

VeriCode includes 5 categories of instructions:

Coding Style & Conventions (9):

• Line length, indentation, variable naming, import order

Logic & Code Patterns (9):

• Number of branches, nesting depth, function complexity, use of specific patterns

Documentation & Commenting (6):

• Docstring style (Google/NumPy/PEP 257), comments, parameter descriptions

Error Handling & Management (4):

• Exception handling, canonical error classes, specific except blocks

Library & API Constraints (2):

• Using specific libraries (pathlib instead of os.path), replacing deprecated APIs

Each instruction includes:

• A category
• A natural-language description for single/multi-turn modes
• Configurable parameters (for example, line_length: 79 or 88)
• Verifier code (returns True/False)

Task: Write a Slack bot that sends notifications about new orders from the CRM. The code needs to be readable for the team — junior devs will be finishing it.

Prompt:

Build a Slack bot in Python that sends notifications about new orders.

Code requirements:
1. Maximum 88 characters per line
2. Use pathlib instead of os/os.path for file handling
3. Document functions with Google-style docstrings
4. Use specific exceptions instead of a generic Exception
5. Maximum 3 branches (if/elif/else) per function

The bot should:
- Read the token from a .env file
- Load order data from a JSON file
- Post a formatted message to the channel

Result:

The model will produce code that technically works, but with a probability of ~47% it will violate at least one of the five requirements. Instruction number 3 (in the middle of the list) will suffer most often because of position bias. Functionality may also degrade by 6-9% — for example, the bot might not handle some file edge cases that it would have handled without the extra instructions.

If you give the same instructions gradually (first a basic bot, then "add the line-length limit," then "add pathlib," and so on), the code will meet the requirements better (~55% success instead of 47%), but functionality will take a bigger hit (a drop of up to 9% instead of 6%).

LLMs generate code while automatically balancing many goals: working code, readability, style, performance. When you add explicit instructions, the model redistributes its attention — part of its "compute budget" goes to meeting the new requirements at the expense of core functionality.

Position bias comes from the transformer architecture: the attention mechanism pays more attention to the beginning and end of the context. Instructions in the middle of the list "get lost" even in short prompts (a few hundred tokens). Single-turn shows primacy bias (the first instruction is followed best), multi-turn shows recency bias (the last one is).

Multi-turn vs single-turn: In single-turn, the model sees the whole picture at once and prioritizes keeping the code working. In multi-turn, each iteration focuses on a local task (add a new requirement), which boosts instruction compliance but more often breaks what used to work — the classic edit distance problem.

The correlation with human preferences shows that users intuitively judge code on both axes — whether it works and its style/patterns. For real-world tasks (BigCodeBench), instruction following is even more important than functionality (a weight of 0.7 vs 0.3); for algorithmic tasks (LiveCodeBench) it's the other way around.

Control Levers

Number of instructions: 1-2 instructions — safe, 3-4 — noticeable regression, 5+ — high risk. For complex tasks, split them across several prompts instead of one prompt with 5+ requirements.

Order of instructions: Put the most important ones at the beginning or end and the secondary ones in the middle. If you're using multi-turn, add critical requirements last (recency bias).

Interaction mode: Single-turn for production code where functionality is critical. Multi-turn for prototypes or when style/patterns matter more than working code.

Instruction parameters: VeriCode supports parameter tuning — you can vary line_length (79/88/120), max_branches (2/3/4), docstring_convention (Google/NumPy/PEP257). Stricter values → higher chance of a violation.

For 1-2 instructions (the safe zone):

Write {task description}.

Code requirements:
1. {instruction_1}
2. {instruction_2}

Task: {details}

For 3+ instructions (watch the functionality):

Option A — Single-turn (preserve functionality):

Write {task description}.

Code requirements (important ones at the edges):
1. {critical_instruction_1}
2. {secondary_instruction}
3. {secondary_instruction}
4. {critical_instruction_2}

Task: {details}

Option B — Multi-turn (maximize instruction following):

// Round 1:
Write {task description}.
Task: {details}

// Round 2:
Add a requirement: {instruction_1}

// Round 3:
Add a requirement: {instruction_2}

// And so on, with critical requirements last

Specific VeriCode instructions (examples):

Style:

• No line longer than {79/88/120} characters
• Use Google/NumPy/PEP257-style docstrings
• Sort imports: standard library → third-party → local

Logic:

• Maximum {2/3/4} branches (if/elif/else) per function
• Nesting depth no more than {2/3} levels
• Cyclomatic complexity ≤ {5/10/15}

Libraries:

• Use pathlib instead of os.path/os/glob/open
• Replace all aliased exceptions (IOError, WindowsError) with the canonical OSError

Error handling:

• Use specific except clauses instead of a generic Exception
• Don't use bare except (except for SystemExit/KeyboardInterrupt)

⚠️ Functional regression: Adding 5 non-functional instructions lowers functionality by 6% on average for real-world tasks and by up to 10% for algorithmic ones. The more instructions, the higher the risk.

⚠️ Low success rate: Even top models satisfy all 5 instructions at once only 40-47% of the time. With 3+ instructions, most models fall below 50%.

⚠️ Position bias: Instructions in the middle of the list (positions 2-3 out of 5) are followed 5-10% worse than those at the beginning or end. The effect shows up even in short prompts.

⚠️ Multi-turn trade-off: Adding instructions gradually raises compliance by 3-8% but amplifies functional regression by 3-5% compared with single-turn.

⚠️ Task specifics: The method was built for Python. Other languages need their own linters and verifiers. Algorithmic tasks suffer more than real-world ones (10% vs 6% regression).

The researchers took 800+ rules from the popular Python linter Ruff, filtered them in three stages and ended up with 30 core instructions. First they removed duplicate and overly narrow rules, then tested on Gemini 2.5 Flash: anything followed 90%+ of the time without losing functionality was thrown out. They kept only the ones where even top models stumble. Each instruction got a deterministic verifier — either linter-based or via AST analysis.

Next they extended two classic benchmarks: BigCodeBench (1,140 real-world tasks) and LiveCodeBench (1,055 algorithmic ones). For each task, an LLM selector (Claude 4 Opus) picked a subset of the 30 instructions — only relevant ones that didn't conflict with each other. That produced over 10,000 checks at the level of individual instructions.

They tested 31 models from 10 families in two modes: single-turn (all instructions at once) and multi-turn (gradually). They measured two metrics: pass@1 for functionality and instruction-level/task-level success for meeting the requirements. A task counted as a success only if ALL instructions were satisfied — hence the exponential drop as the number of requirements grows.

Why these conclusions: Multi-turn showed a systematic pattern — better IF, worse functionality. That's no accident but an architectural property: in multi-turn, the model focuses on a local task at each iteration (add a new requirement) without seeing the full picture as it does in single-turn. Position bias reproduced on both benchmarks with different shapes: a U-shape on real-world tasks (lost-in-the-middle), a linear trend on algorithmic ones, but everywhere — the middle does worse than the edges.

The link to human preference: They correlated the results with 800K+ votes from the LMArena coding subset. The correlation peaks with a mix of metrics (40-70% weight on IF), not with pure functionality or pure IF. And for real-world tasks IF matters more (optimum at 70%), while for algorithmic ones functionality does (optimum at 30%). This explains why Copilot Arena rankings correlate weakly with classic benchmarks: those benchmarks measure only half of what users value.

Vibe Checker: Aligning Code Evaluation with Human Preference
Ming Zhong, Xiang Zhou, Ting-Yun Chang and team (Google DeepMind, UIUC, USC)
arXiv:2025.XXXXX (pre-print)

The VeriCode taxonomy and verifiers will be released publicly.
Ruff linter: https://docs.astral.sh/ruff/rules
BigCodeBench: https://openreview.net/forum?id=YrycTjllL0
LiveCodeBench: https://openreview.net/forum?id=chfJJYC3iL
LMArena coding leaderboard: https://lmarena.ai/leaderboard/text/coding