TL;DR
The Failure-Aware Framework is a system for choosing a code-improvement method when basic progressive prompting (step-by-step generation: requirements → design → tests → code) falls short. The core idea: different failure types need different fix strategies: Self-Critique (self-review), Multi-Model Collaboration (different models for design and code) or RAG-Assisted (documentation in the context).
Progressive prompting delivers 96.9% task completion vs. 80.5% for a direct prompt, but 32% of projects still end up unfinished. The failures fall into 4 types: logic (missing validation), integration (API not wired up), domain (unknown medical rules), infrastructure (Docker config not set up). Self-Critique works great on logic failures (they're visible in code review), but gives 0% improvement on integration failures: the model doesn't know how to configure RabbitMQ no matter how many times it reviews the code. RAG-Assisted solves this by dropping RabbitMQ tutorials right into the context.
The decision framework is a "failure type → method" map: for logic failures use Self-Critique (fast, 138 minutes); for integration/domain/infrastructure use RAG-Assisted (193 minutes, 31.5 min per percentage point of improvement); for critical tasks use Multi-Model (279 minutes, but 100% on 5 of 6 projects). It's empirically proven: not every method works everywhere, and the choice depends on what kind of gap you're dealing with.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Baseline: Progressive Prompting
STEP 1: Requirements analysis → task list
STEP 2: Architecture design → project structure
STEP 3: Test specification → test scenarios
STEP 4: Code generation → implementation
Each step is a separate prompt in the same session.
Enhancement 1: Self-Critique (self-review)
STEPS 1-4: Progressive Prompting
STEP 5: "Review the code. Are all requirements met? What's missing?"
STEP 6: The LLM finds the gaps → generates fixes
STEP 7: Repeat 5-6 until the LLM says "everything's done"
An iterative loop in the same session.
Enhancement 2: Multi-Model Collaboration
In ChatGPT (GPT-5):
STEP 1: Requirements analysis
STEP 2: Architecture design → detailed specification
In Claude (Sonnet 4.5):
STEP 3: Paste in the specification from GPT
STEP 4: Generate code from the design
Switching between chats: you carry the specification over by hand.
Enhancement 3: RAG-Assisted (documentation in the context)
PREP: Find documentation/examples (official docs, GitHub projects)
STEPS 1-4: Progressive Prompting + add to the prompt:
"Using these examples and documentation [paste], implement the requirements"
Copy-paste the documentation into every relevant prompt.
Example
Task: You're building a WhatsApp ordering bot for a coffee shop. Progressive prompting generated the basic code (menu, cart, commands), but couldn't integrate Stripe payments: the code just has a stub # TODO: payment integration.
Diagnosing the failure type
You look at the code: the cart logic is there, validation is there, but the external API (Stripe) isn't wired up. That's an External Integration Failure.
According to the decision framework: - 1st choice for integration failures: RAG-Assisted - 2nd choice: Multi-Model - 3rd choice: Self-Critique (doesn't work on these failures)
Applying RAG-Assisted
Prompt:
Using the official Stripe documentation for Python and the examples below,
finish the create_payment() function for paying for an order in the WhatsApp bot.
[paste the Quick Start page from stripe.com/docs/api]
[paste a checkout.Session.create() example from Stripe's GitHub samples]
Requirements:
- Create a payment for the cart total
- Return a payment link to the user
- Handle the payment confirmation webhook
Current function code:
[paste the stub from the generated code]
Result:
The model fills the stub with working code: the stripe import, setting stripe.api_key from STRIPE_SECRET_KEY, a call to stripe.checkout.Session.create() with the right parameters (line_items, currency="usd", mode="payment"), and handling of session.url. If progressive prompting didn't know how to work with a specific API, RAG gives it an exact template.
Why it works
The LLM's weakness: The model is trained on general patterns but doesn't know the specifics of particular APIs, frameworks and domain rules. When it sees "integrate Stripe," it generates plausible but broken code, because it never saw the current 2024 documentation in its training data.
The LLM's strength: It's excellent at adapting templates to a new context. Give it a checkout.Session.create() example with parameters and it immediately gets the logic and plugs in the right values from your task. This isn't blind copying: the model understands the structure and transfers the pattern.
How the methods work: - Self-Critique uses the LLM's ability to analyze code as text. It works when the error is visible: a missing function, forgotten validation. It doesn't work when outside knowledge is needed (how to configure RabbitMQ). - Multi-Model uses model specialization: GPT-5 is stronger at system design, Claude at code generation. Splitting the work reduces ambiguity in the requirements. - RAG-Assisted feeds in concrete examples and official patterns, turning "figure out how the API works" into "adapt this working example."
Control levers: - Number of Self-Critique cycles: you can stop after 2-3 iterations instead of "until it says done," which saves time once it's clear the method isn't helping - Models in Multi-Model: you can swap GPT-5 for GPT-4o (cheaper) for the design step if the project is simple - Depth of RAG materials: for simple tasks a Quick Start is enough; complex ones need GitHub examples and best practices - Type of documentation in RAG: official docs (accuracy) vs. Stack Overflow (practicality) vs. GitHub (real code)
Prompt template
1. Progressive Prompting (baseline)
Step 1: Analyze the requirements and make a task list for the project: {project_description}
---
Step 2: Based on the task list, design the architecture: code structure, components, API, data models.
---
Step 3: Write a test specification for each component: which scenarios to test, edge cases.
---
Step 4: Implement the project code following the architecture and tests from the previous steps.
Placeholders: {project_description}: what you'd normally type into the chat: "A WhatsApp bot for a coffee shop with a menu, cart and payments."
2. Self-Critique (self-review)
[After progressive prompting has generated the code]
Review the generated code carefully:
- Are all the tasks from step 1 implemented?
- Which requirements are missing or only partly done?
- Where are the stubs or TODOs in the code?
List the gaps, then fix them.
[Repeat until the LLM answers "everything is done" or you see it's no longer improving]
3. Multi-Model Collaboration
In ChatGPT (GPT-5):
Step 1: Analyze the requirements for the project: {project_description}
---
Step 2: Design a detailed architecture:
- Component structure
- API endpoints / interfaces
- Data models
- Integration patterns for external services
Produce a complete specification to hand off to another developer.
In Claude (Sonnet 4.5):
Implement the code according to the following architecture specification:
[paste the specification from GPT]
Follow the design exactly; use the specified patterns and structure.
4. RAG-Assisted (documentation in the context)
Using the documentation and examples below as reference, implement {task}:
[paste the official API/framework documentation]
[paste code examples from GitHub / tutorials]
Requirements:
{list_of_specific_requirements}
Current code (if any):
{code_with_stubs}
Placeholders:
- {task}: "Stripe payment integration for a WhatsApp bot"
- Documentation: copy from the Quick Start and API Reference pages
- Examples: search GitHub for "{library} example" and take the relevant chunks
- {list_of_specific_requirements}: specifics: "create a payment for the cart total, return a link, handle the webhook"
- {code_with_stubs}: whatever progressive prompting already generated with TODOs
Where to find RAG materials: - Official docs: stripe.com/docs, developers.facebook.com/docs/whatsapp - GitHub: search "{library name} example" → check the Issues and Wiki - Tutorials: dev.to, Medium; filter by recency (2023-2024)
Decision Framework: what to use when
1. Diagnose the failure type in the generated code:
Logic (Local Logic):
- Missing function
- Forgotten validation
- Incomplete business logic
→ 1st choice: Self-Critique
→ 2nd choice: RAG-Assisted
Integration (External Integration):
- API not wired up
- Service not configured
- External SDK used incorrectly
→ 1st choice: RAG-Assisted
→ 2nd choice: Multi-Model
Domain (Domain Knowledge):
- Industry-specific rules
- Business processes
- Regulatory requirements
→ 1st choice: RAG-Assisted
→ 2nd choice: Multi-Model
Infrastructure:
- Docker configs
- CI/CD pipeline
- Deployment
→ 1st choice: RAG-Assisted
→ 2nd choice: Multi-Model
2. Apply the 1st choice. If it doesn't help → 2nd choice.
3. For critical tasks (production code) go straight to Multi-Model:
slower but more reliable (100% in 83% of cases).
How to diagnose: - Open the generated code - Find the TODOs, stubs, missing pieces - Ask yourself: "Is this visible in the code, or does it need outside knowledge?" - Visible (missing function, no check) → Logic → Self-Critique - Needs knowledge (how the API works) → Integration/Domain → RAG-Assisted
Limitations
⚠️ Requires a diagnosis: You need to identify the failure type correctly. Apply Self-Critique to an integration problem and you'll burn 138 minutes for nothing (0% improvement in the study).
⚠️ Multi-Model = two subscriptions: You need access to GPT-5 (or GPT-4) and Claude Sonnet 4.5. If you only have one chat, use RAG-Assisted as the all-purpose method.
⚠️ RAG means finding materials: Effectiveness depends on the quality of the documentation. If the API is obscure and examples are scarce, the method delivers less. In the study, RAG couldn't handle project P10 (complex inventory logic): no ready-made examples, only generic CRUD patterns.
⚠️ Code-specific: The framework was built for code generation, but the principles are universal: Self-Critique for errors fixable through review, RAG for tasks that need outside knowledge, Multi-Model for critical cases.
⚠️ Time vs. accuracy: Self-Critique is the fastest (138 min on average) but doesn't work everywhere. Multi-Model is the slowest (279 min) but the most reliable. RAG-Assisted is the sweet spot (193 min, best efficiency at 31.5 min/percentage point).
Why the method works: the study's logic
The researchers took 25 real projects from GitHub (e-commerce, booking, microservices, infrastructure) and compared direct prompting (one "write the code" prompt) with progressive prompting. Result: progressive delivered 96.9% completion vs. 80.5% for direct (a large effect, Cohen's d=1.63, p<0.001), but 8 of 25 projects (32%) were left unfinished, with 19 tasks not implemented.
The team classified these 19 gaps into 4 types by the nature of the missing functionality (not syntax, but what exactly was left out): logic errors (missing inventory validation), integration (RabbitMQ not configured), domain (medical rules unknown), infrastructure (missing Docker configs). They then picked 6 representative projects (covering all 4 types) and ran 3 enhancement methods on each = 18 experiments.
The surprise: Self-Critique gave exactly 0% improvement on 3 of 6 projects (P8, P18, P25): the ones with external integrations and domain knowledge. No amount of iterative code review lets the model conjure up how to configure RabbitMQ out of thin air. But on projects P2 and P10 (logic gaps: a forgotten API endpoint, skipped validation), Self-Critique delivered 100% in 115-155 minutes. That revealed the method's clear boundary: it only works on code-reviewable failures, the ones you can see when reviewing the code.
Multi-Model hit 100% on 5 of 6 projects (83%) but was never the fastest: 279 minutes on average vs. 193 for RAG and 138 for Self-Critique. RAG-Assisted showed the best versatility: 100% on every integration/domain/infrastructure project, plus top efficiency at 31.5 minutes per percentage point of improvement vs. 47.4 for Multi-Model. The paradox: RAG and Multi-Model both averaged 99.2%, but RAG won 4 of 6 projects on speed.
The practical insight: There's no universal "best method." Self-Critique is a fast niche tool (it works on only 1 of 4 failure types). RAG-Assisted is the all-purpose workhorse (works everywhere with top efficiency). Multi-Model is insurance for critical tasks (the most reliable, but expensive in time). The decision framework is an empirical map: each failure type gets a method with proven effectiveness, no trial and error.
Resources
Failure-Aware Enhancements for Large Language Model (LLM) Code Generation: An Empirical Study on Decision Framework
Jianru Shen, Zedong Peng, Lucy Owen, University of Montana
Experiment data: https://doi.org/10.5281/zenodo.17637008