Zero-Shot (baseline):

Task → Code

Zero-Shot CoT (can backfire):

Task + "Let's think step by step" → Free-form reasoning → Code

⚠️ The reasoning may be logical yet functionally wrong

Self-Planning (structured):

STEP 1: Decomposition → list of subtasks
STEP 2: Execution plan → implementation order
STEP 3: Code → final solution

(3 separate calls with few-shot examples)

SCoT — Structured CoT (structured):

Slot 1: Problem analysis → what needs solving
Slot 2: Algorithm → which structures (loops/branches)
Slot 3: Implementation → code

(1 call with a template and 3 examples)

Reasoning-CoT (deep):

DeepSeek-R1 model → Long reasoning with self-verification → Code

(~2000-7000 reasoning tokens)

Task: You're building a Slack bot that collects sales leads. You need a phone number validation function — it should accept +1 and 1 prefixes and various formats (with spaces, parentheses, dashes). If you just ask for the code, the model may skip edge cases.

Prompt (SCoT):

Write a function that validates a US phone number.

Use this structure:

**Problem analysis:**
- Which number formats need to be supported?
- What edge cases could come up?

**Algorithm:**
- Which checks should be performed?
- In what order?
- Which data structures?

**Implementation:**
[your code with comments]

Requirements:
- Support +1 and 1 at the start
- Ignore spaces, dashes, parentheses
- Length check (10 digits after the country code)

Result: The model will explicitly list every format (+1-XXX-XXX-XXXX, 1(XXX)XXX-XXXX, etc.), enumerate the edge cases (empty string, letters, numbers that are too short), describe the algorithm (normalize → check prefix → check length), and output code that handles all of them. The structure forces it to think through the details before writing any code.

The LLM's weakness: The model generates text by plausibility patterns, not by functional correctness. Free-form reasoning ("think step by step") can be logically coherent yet functionally wrong — the model isn't checking whether the code will run, it's checking whether the reasoning sounds sensible. It gets especially dangerous when the model "confidently" writes about handling edge cases and then leaves them out of the code.

The LLM's strength: Models are great at following structured instructions and filling in explicit slots. When you hand it a rigid template with concrete questions ("Which edge cases?", "Which checks?"), the model has to answer each one. The template works like a checklist — if the model can't fill a slot, it won't move on to the code.

How the method uses this: SCoT and Self-Planning give the model a predefined thinking structure instead of free-form musing. That cuts off irrelevant reasoning and focuses it on what matters functionally: inputs, algorithm, edge cases, order of operations. The harder the task, the bigger the effect (BigCodeBench: +11% for SCoT vs Zero-Shot).

Control knobs: - Number of slots (3-5 is optimal) → more slots = more detailed plan, but slower generation - How specific the slot questions are ("Which checks?" instead of "Think through the solution") → the more specific, the fewer hallucinations - Examples (few-shot) → 3 examples give +5-7% accuracy over zero-shot with the same template - Quality of the CoT generator → if you use a different model to generate the reasoning, a strong model (GPT-5) gives +7.5% over a weak one (7B)

SCoT (Structured CoT) — for complex tasks

{task_description}

Use this structure:

**Problem analysis:**
- What needs to be solved?
- What are the inputs/outputs?
- What are the edge cases?

**Algorithm:**
- Which steps to perform?
- Which data structures to use?
- Which checks are needed?

**Implementation:**
[code with comments]

{additional_requirements}

When to use it: - {task_description} — a coding task where edge cases, data types, or performance matter - {additional_requirements} — specific constraints (programming language, libraries, output format)

Self-Planning — for breaking work into subtasks

{task_description}

Step 1 — Decomposition:
Break the task into subtasks. List each one.

Step 2 — Plan:
Describe the order in which to implement the subtasks and how they connect.

Step 3 — Implementation:
Write the code according to the plan.

{additional_requirements}

When to use it: - The task is complex and it's unclear where to start - You need a system architecture (several modules/functions) - Implementation order matters

🚀 Quick start — paste into a chat:

Here's an SCoT template for structured code generation. Adapt it to my task: [your task].
Ask me questions to fill in the "Problem analysis", "Algorithm", and "Implementation" slots.

[paste the template above]

The LLM will ask about the specifics of your task (inputs, constraints, edge cases), because it needs to fill the template's slots. It'll take the structured-thinking pattern and fit it to your problem.

⚠️ Overkill for simple tasks: If the task is trivial ("write a function that adds two numbers"), structured CoT just burns extra tokens with no accuracy gain. For simple code, Zero-Shot is faster and cheaper.

⚠️ Depends on the quality of the reasoning generator: If you use a small model (7B) to generate CoT reasoning that you then feed to another model, the effect will be weaker than with a strong generator (GPT-5). The gap is up to 7.5% Pass@1.

⚠️ Not a cure-all for weak models: Structured CoT gives the biggest boost to mid-sized models (GPT-3.5: +12%). Very small models (<7B) don't always extract information from the reasoning — they may follow the structure on paper but generate irrelevant content.

⚠️ Code-specific: The study is about code generation. For other tasks (writing, analysis, creative work) the effects may differ. But the principle "structure > free-form musing" carries over.

The team tested 5 CoT paradigms on 6 models (from 7B to 480B parameters), 6 Python benchmarks (from the simple MBPP to the hard BigCodeBench), and 12 programming languages. They measured Pass@1 (the share of tasks solved correctly on the first try) and token cost.

A clever design: To understand why CoT works, they used an information-theoretic approach — the reasoning _C_ was treated as an information channel between the task _X_ and the code _Y_. CoT efficiency = _how much useful information_ the reasoning carries / _how many tokens_ it uses. That explained why structured CoT is more efficient: high information density with low noise.

Controlled experiments: For RQ3 (dependence on model size) they compared pairs of models from the same family (GPT-3.5 vs GPT-5, Qwen 7B vs 32B) with identical prompts. That let them isolate asymmetric cases — where the big model succeeds and the small one fails with the same CoT. A pattern emerged: small models systematically fail on data types and edge cases, even when the reasoning is correct.

Surprising finding: Zero-Shot CoT ("Let's think step by step") often hurt results compared to direct generation. Digging into the examples, they saw the model generating logically coherent reasoning that looks right but contains functional bugs. For instance, it writes "let's handle the empty-list edge case", but the code has no such check. That's hallucinated reasoning — plausible text with no functional value.

Practical takeaway: Structured CoT (Self-Planning, SCoT) reaches 58-60% average accuracy at ~200-700 tokens, while deep Reasoning-CoT gets 59-61% at ~2000-7000 tokens. In efficiency terms: structured approaches deliver 85-95% of the accuracy of deep reasoning for ~10% of the tokens. For production scenarios with cost/latency budgets, structured CoT is the better deal.

Context: The authors tested the SCoT prompt with 3 few-shot examples. Here's one of the examples showing the structure:

Question: Write a function to find the similar elements from the given two tuple lists.

**Problem Analysis:**
- Input: two tuples
- Output: a set of common elements
- We need to find elements that appear in both tuples

**Algorithm Design:**
- Convert tuples to sets
- Use set intersection to find common elements
- Return the result as a set

**Implementation:**
def similar_elements(test_tup1, test_tup2):
    res = set(test_tup1) & set(test_tup2)
    return res

💡 Adapting it for non-technical tasks:

The principle "structured slots > free-form musing" isn't just for code. It applies anywhere functional correctness matters, not just plausible-sounding text.

Example — a marketing plan for a new product:

Product: {description}
Target audience: {who}

Use this structure:

**Situation analysis:**
- What pain point does the product solve?
- Who are the competitors and how are we different?
- What are the barriers to purchase?

**Positioning strategy:**
- What's the key message?
- Which communication channels?
- What proof of value?

**Tactical plan:**
- Which activities in the first month?
- Which metrics do we track?
- What's the budget per channel?

Without structure, the model can hand you a plan that "sounds convincing" but won't work in practice (no metrics, unrealistic budgets, channels that miss the audience). The structure forces it to check the functional side: measurability, feasibility, audience fit.

🔧 Technique: add an "Edge cases" slot → catch the corner cases

The study showed that models often skip edge cases even when they mention them in the reasoning. An explicit "Edge cases" slot that requires the model to list each one and show the code for it reduces the misses.

Modified SCoT:

{task}

**Problem analysis:**
[same as the original]

**Edge cases (required):**
List every edge case and, for each one, write how you'll handle it in the code.

**Algorithm:**
[same as the original]

**Implementation:**
[code with explicit handling of every edge case from the list above]

The effect: the model can't skip an edge case in the code if it listed it in a separate slot. The slot works as a self-check checklist.

🔧 Technique: use case for small models (7B-13B) → compensating for limited capacity

The study showed that small models gain the most from structured CoT (+12% for GPT-3.5 vs +1.5% for GPT-5). If you work with a local or cheap small model, structured CoT is essential.

In practice: For small models, make the slots even simpler and more specific:

{task}

**Step 1 — What comes in:**
List all input parameters and their types.

**Step 2 — What goes out:**
Describe the result and its format.

**Step 3 — What to check:**
List all checks (types, lengths, empty values).

**Step 4 — Code:**
[implementation strictly following steps 1-3]

The weaker the model, the more atomic the slots should be — one thought = one slot.

Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis

Methods covered in the study: - Chain-of-Thought (Wei et al. 2022) - Zero-Shot CoT (Kojima et al. 2022) - Self-Planning (Jiang et al. 2024) - Structured CoT — SCoT (Li et al. 2025) - Multilingual SCoT — MSCoT (Jin et al. 2025)

Models tested: - Qwen2.5-Coder (7B/32B) and Qwen3-Coder (30B/480B) — Alibaba DAMO - GPT-3.5-Turbo and GPT-5 — OpenAI - DeepSeek-R1 — for Reasoning-CoT

Benchmarks: MBPP, HumanEval, OpenEval, MHPP, CodeHarmony, BigCodeBench, HumanEval-XL (12 languages)

Authors: Naizhu Jin, Zhong Li (corresponding), Guang Yang, Tian Zhang, Qingkai Zeng Affiliation: State Key Laboratory for Novel Software Technology, Nanjing University; State Key Laboratory of Blockchain and Data Security, Zhejiang University