QoT (Questions-of-Thoughts): Multi-Step Self-Checking for Complex Tasks

All steps run in a single prompt:

STEP 1: Decomposition
        Take goal G → break it into an ordered list of steps S1, S2...Sn

STEP 2: Q&A verification (repeated for each step Si)
        → Generate verification questions Qi,1...Qi,m for step Si
        → Answer each question
        → Add the answers to the "working memory" TP
        Repeat for the next step

STEP 3: Final output
        Use all the accumulated "working memory" TP
        → Build the final answer R

QoT works where you need systematic completeness: multi-part tasks where it's easy to miss something. It's not a fit for simple, unambiguous requests.

Task: Ethan is launching a barbershop franchise and wants to build an onboarding system for new franchisees — so a partner covers everything needed in the first 30 days and doesn't "get lost". He needs a full playbook with modules, checkpoints and owners.

Prompt:

You are an expert in franchise operating systems. 
Your task is to develop an onboarding playbook for a new barbershop franchisee covering the first 30 days.

Follow the QoT algorithm strictly:

**STEP 1. Decomposition**
Break the onboarding into an ordered sequence of modules. 
Each module is a separate block that logically depends on the previous one.
Output a numbered list of modules.

**STEP 2. Self-check for each module**
For each module from Step 1, do the following:
- Write 5–8 verification questions in the form "What if...", "How is ... handled", "Who is responsible for...", "What happens when..."
- Answer each question specifically
- Record the answers as the module's "working decisions"

Do this for all modules in sequence.

**STEP 3. Final playbook**
Based on all the working decisions accumulated in Step 2, write the final onboarding playbook:
- Timeframe for each module
- Action checklist
- Responsible parties (franchisor / franchisee)
- Checkpoints and success criteria

Task: develop an onboarding playbook for a new barbershop franchisee covering the first 30 days.

Result: The model first shows the structure of onboarding modules (legal paperwork, training on standards, equipment setup, marketing launch, grand opening, etc.). Then, for each module, a list of questions with answers: who sends the contract, what to do if the supplier delays the equipment, how to verify a barber completed training. At the end — a coherent, complete playbook with no gaps like "wait, who calls customers in the first week?"

The problem: When you ask a model to "write a playbook", it builds the text linearly — as it generates. If it didn't "glance" at the section about early-stage failures, it won't write it. The result is a pretty but leaky document.

The model's strength: Models are good at answering specific questions. A question sets a frame — and the model finds the answer inside that frame. No question, no frame, no answer.

How QoT uses this: The method forces the model to generate its own questions before giving an answer. It's a built-in checklist that's built dynamically for each specific step — not a generic "think carefully", but targeted "what if the client hasn't signed the acceptance form". The accumulated answers create context the model can no longer "forget" — it's right in front of it during final generation.

Control levers: - Number of questions per step → for simple tasks use 3–4 instead of 7–8 to save tokens - Question focus → add a theme ("especially around data security", "around errors and failures") — the model will generate questions in that direction - Drop the decision logging → if you only want structure without detail, remove the "record as working decisions" instruction - Number of steps → the more complex the task, the more modules you should spell out explicitly in Step 1

You are an expert in {subject_area}.
Your task is to {end_goal}.

Follow the QoT algorithm strictly:

**STEP 1. Decomposition**
Break the task into an ordered sequence of modules/stages.
Each element is a separate logical block.
Output a numbered list.

**STEP 2. Self-check for each element**
For each element from Step 1:
- Write {number_of_questions} verification questions: 
  "What if...", "How is ... handled", "Who is responsible for...", "What happens when..."
  [optional: focus on {specific_topic}, e.g. errors, security, areas of responsibility]
- Answer each question specifically
- Record the answers as working decisions

Repeat for all elements in sequence.

**STEP 3. Final result**
Based on all the working decisions accumulated in Step 2, 
prepare: {final_answer_format}

Task: {detailed_task_description}

What to fill in: - {subject_area} — operations management, product marketing, HR, law - {end_goal} — create a playbook / develop a strategy / write a spec - {number_of_questions} — 5–8 for complex tasks, 3–4 for medium ones - {specific_topic} — optional, but it sharpens the questions toward the angle you need - {final_answer_format} — checklist, table, step-by-step playbook, document - {detailed_task_description} — specifics: who, what, for whom, what constraints

🚀 Quick start — paste into the chat:

Here's a template for the QoT (Questions-of-Thoughts) method. 
Adapt it to my task: [your task].
Ask me questions to fill in all the fields.

[paste the template above]

The LLM will clarify the subject area, the final format and the focus for the verification questions — because without that it can't properly set up Step 2, which determines the quality of the final output.

⚠️ Weak models: On low-powered models (GPT-3.5, Mistral 7B), the self-check can be shallow — questions get generated as a formality, without real depth. The method shines on strong models.

⚠️ Simple tasks: If the task is unambiguous and easy, QoT is overkill. The questions will be trivial and the answer will get longer with no gain in quality.

⚠️ The "over-engineering" effect: On some tasks with large models, the opposite happens — the model starts overcomplicating where a simple solution is enough. The authors saw this with file systems: large models proposed excessive architecture.

⚠️ Tokens: The method is much more token-hungry than a regular request. Cost grows multiplicatively: number of steps × questions per step × answer length. For long tasks, budget your context.

⚠️ Doesn't guarantee correctness: QoT improves completeness and structure, but it doesn't eliminate hallucinations. You still need to verify the facts.

Researchers from Taiwan took four models of different sizes (from a tiny 3B to a hefty 70B from the Llama family) and gave each three kinds of backend code generation tasks: a room-booking API, a messaging system and a file manager. Each task deliberately required several interconnected modules — so it would be easy to "forget" part of it.

They compared three modes: no hints, Chain-of-Thought ("think step by step") and QoT. They judged not "does the code technically run" but quality on four criteria: scalability, completeness, modularity, security — on a 1–4 scale. The judge was a strong model (Gemini Pro), which reviewed the work blind — without knowing which method generated it.

An interesting design detail: to keep the judge from "drifting" in its scores from round to round, it was always shown the results of the previous round — for calibration. The outcome was mixed: for large models QoT consistently helped on the API and messaging tasks, but on the file system task it dropped. The authors put this down to "over-engineering": a large model with a big planning context starts overcomplicating where the task doesn't call for it. The most unexpected result: the small 3B model with QoT beat itself without QoT by ~100% in quality — meaning the method literally doubled a weak model's output under the right conditions.

🔧 Domain-focused questions

By default, the questions are generated broadly. Add a focus and Step 2 gets sharper:

For each element, write 6 questions
strictly focused on:
— failure points and edge cases
— areas of responsibility (who exactly)
— risks to the customer

Use it for tasks where a specific angle matters: process audits, legal contract review, deal risk analysis.

🔧 QoT + a critic role

After Step 3, add one more layer:

**STEP 4. Critical review**
Read the final result from Step 3 as a skeptical investor/lawyer/auditor.
Find: what's missing, what's worded vaguely, where the risks are.
Suggest specific edits.

QoT + an external critique = double insurance for completeness. Works especially well for pitches, contracts and strategies.

Paper title: Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chain

Repository with prompts and examples: https://github.com/knyliu/questions-of-thoughts

Authors: Yen-Ku Liu, Yun-Cheng Tsai — Department of Technology Application and Human Resource Development, National Taiwan Normal University, Taipei, Taiwan

Contact: [email protected]