Apply to my task
arXiv:2601.14152 ★ 85 Jan 20, 2026

CQO vs QOC: Why the Order of Prompt Elements Swings Accuracy by 15%

Key result

Shuffle your prompt elements and lose 15% accuracy. The cause isn't the model's training, it's the architecture. The CQO format (context→question→options) gives 15% more accurate answers on multiple-choice tasks than QOC (question→options→context). The trick: decoder models (GPT, Claude, LLaMA) use a one-way attention mask. Each token sees only what came BEFORE it, never what comes AFTER. Put the options BEFORE the context and they physically can't use the information in it. The model picks blind.

01

TL;DR

In multiple-choice tasks, the order of prompt elements is critical. The CQO format (context → question → options) is 14.7% more accurate than the reverse order, QOC (question → options → context). The gap holds for every decoder-only model (GPT, Claude, LLaMA, Qwen, Gemma) on every task. The cause isn't training or forgetfulness, it's architecture.

Decoder-only models use a causal mask: each token sees only the tokens before it, never the ones after. That creates an information blockade: in QOC the answer options come BEFORE the context, so the model physically can't use the context while processing the options. It's forced to pick an answer blind, relying only on the options and the question. The researchers showed that QOC accuracy = accuracy with no context at all (question + options only).

The fix: always put the context BEFORE the answer options. If your structure demands QOC, use the QOCO workaround: repeat the options after the context. The repeated options will "see" the context, and the model can make an informed decision. This lifts QOC accuracy by 8.2% without touching the model.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Problem Blueprint

Decoder-only model with a causal mask:

CQO (works well):
[Context] → [Question] → [Options A/B/C/D]
          ↓              ↓
    Options SEE the context through the causal mask

QOC (works poorly):
[Question] → [Options A/B/C/D] → [Context]
                    ↓               ✗
         Options DON'T SEE the context (causal mask blocks it)

QOCO (workaround):
[Question] → [Options₁] → [Context] → [Options₂]
                                    ↓
                         Options₂ SEE the context
03

Example in Practice

Task: You're picking an agency to build a website. You've got three proposals with different approaches. You need to choose the best fit for the project requirements.

Prompt (correct: CQO):

**Project context:**
Online store for a furniture company. Priorities: fast loading (customers are often on mobile), a simple admin panel for non-technical managers, integration with QuickBooks. Budget $8,000, timeline 2 months. Stability matters more than creativity.

**Question:**
Which proposal best fits the requirements?

**Options:**
A. Agency #1: React + headless CMS, custom design, $10,000, 3 months
B. Agency #2: WordPress + WooCommerce, ready-made theme + customization, $7,000, 1.5 months
C. Agency #3: Laravel + Vue.js, bespoke architecture, $9,000, 2.5 months
D. Agency #4: Squarespace + external cart, visual editor, $5,500, 1 month

Answer with the option letter and explain your choice in terms of the project priorities.

Result:

The model will analyze each option against the requirements (speed, admin simplicity, budget, timeline, stability) and pick the best fit. The answer will be based on matching the proposals' characteristics to the criteria in the context.

Prompt (wrong: QOC):

**Question:**
Which proposal best fits the requirements?

**Options:**
A. Agency #1: React + headless CMS, custom design, $10,000, 3 months
B. Agency #2: WordPress + WooCommerce, ready-made theme + customization, $7,000, 1.5 months
C. Agency #3: Laravel + Vue.js, bespoke architecture, $9,000, 2.5 months
D. Agency #4: Squarespace + external cart, visual editor, $5,500, 1 month

**Project context:**
Online store for a furniture company. Priorities: fast loading (customers are often on mobile), a simple admin panel for non-technical managers, integration with QuickBooks. Budget $8,000, timeline 2 months. Stability matters more than creativity.

Answer with the option letter and explain your choice in terms of the project priorities.

Result:

The model will pick an option based on general patterns ("WordPress is popular for online stores") rather than the specific project requirements. The context is effectively ignored: accuracy is the same as if it weren't there at all.

Workaround (QOCO: repeating the options):

**Question:**
Which proposal best fits the requirements?

**Options:**
A. Agency #1: React + headless CMS, custom design, $10,000, 3 months
B. Agency #2: WordPress + WooCommerce, ready-made theme + customization, $7,000, 1.5 months
C. Agency #3: Laravel + Vue.js, bespoke architecture, $9,000, 2.5 months
D. Agency #4: Squarespace + external cart, visual editor, $5,500, 1 month

**Project context:**
Online store for a furniture company. Priorities: fast loading (customers are often on mobile), a simple admin panel for non-technical managers, integration with QuickBooks. Budget $8,000, timeline 2 months. Stability matters more than creativity.

**Choose from:**
A. Agency #1  |  B. Agency #2  |  C. Agency #3  |  D. Agency #4

Answer with the letter and explain via the priorities.

Result:

The repeated options come AFTER the context, so the model can match them against the requirements. Accuracy rises by 8% compared with QOC without the repeat.

04

Why It Works

The decoder-only LLM weakness: The causal mask is an architectural constraint. Each token is generated based only on the tokens before it. That prevents "cheating" (the model can't peek into the future during training), but it creates an information blockade with certain prompt orders.

What happens in QOC: When the model processes the answer options, the context doesn't yet exist in its "field of view". The internal representations of the options (hidden states) are formed WITHOUT any information from the context. Even if the final answer token can "see" the whole context, it's already too late: the option representations are locked in, and the model can't recompute them. Comparing the options to the context has to happen at the very last step, using "context-blind" option representations.

What happens in CQO: The context is processed first. When the model reaches the options, they can "see" all the preceding context through the attention mechanism. The option representations are formed WITH the relevant information. The model compares the options against the evidence instead of guessing.

Experimental proof: The researchers measured gradient attribution (how much the context influences the prediction): in CQO the context contributes 79.7% of the answer, in QOC only 33.5%. What's more, QOC accuracy = accuracy with no context at all (QO format). The context is structurally out of reach.

Modulating factors:

  • Context length: The longer the context, the more information is lost in QOC. LogiQA (~70 words): gap 6.2%; RACE-H (~305 words): gap 20.8%.

  • Position of the correct answer: Option D (closer to the context) suffers less (gap 9.9%) than option A (gap 22.4%). The farther an option sits from the context, the lower the odds of a correct answer in QOC.

Why the QOCO workaround works: The repeated options come AFTER the context, so they "see" it through the causal mask. The model uses the second, context-aware option representations for the final choice.

05

Prompt Template

{context}

{question}

{options}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}

{answer_instructions}

How to fill it in: - {context}: the information needed for the decision (requirements, criteria, facts, data) - {question}: the wording of the choice task - {options}: the options to choose from, labeled with letters - {answer_instructions}: the answer format (for example: "Answer with the letter and explain")

Critical: Context ALWAYS goes first, options go last.

Workaround for when you need QOC:

{question}

{options_first_time}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}

{context}

Choose from:
A. {short_option_A} | B. {short_option_B} | C. {short_option_C} | D. {short_option_D}

{answer_instructions}

The second mention of the options can be brief (just the letter + a keyword). What matters is that they come AFTER the context.

06

Limitations

⚠️ Applies only to decoder-only models: Encoder-only (BERT-like) and encoder-decoder (T5-like) models don't have this problem: they use bidirectional attention, so every token sees every token. The effect shows up only in GPT, Claude, LLaMA, Qwen, Gemma and other decoder-only models.

⚠️ Long contexts suffer more: The more information in the context, the more of it gets lost in the QOC format. Short contexts (under 100 words) show a 6-7% gap, long ones (200-300 words) up to 25%.

⚠️ The correct answer's position matters: If in QOC the correct answer is the last option (closer to the context), accuracy drops less (gap 10%). If it's the first option (farther from the context), it drops more (gap 22%).

⚠️ QOCO is a partial fix: Repeating the options lifts accuracy by 8% but doesn't fully close the gap (CQO is still 6-7% more accurate). Use QOCO only if the prompt structure strictly requires QOC (for example, a fixed test format).

07

How They Tested It

A team from POSTECH took 21 decoder-only models (LLaMA 3, Qwen 2.5/3, Gemma 2, from 0.5B to 9B parameters) and tested them on four multiple-choice benchmarks: LogiQA (logic), SciQ (science), RACE-M/H (reading). Each task was prompted in two formats, CQO and QOC, and accuracy was measured.

The result was striking: CQO consistently beat QOC by 14.7% on average, and the gap was the same across all models: base and instruct, small and large. That pointed to a fundamental cause unrelated to training or size.

Testing the hypotheses: The researchers systematically tested three explanations:

  1. Bias in the data? They compared base and instruct models (instruct models saw more CQO examples): the gap was identical. They added 5-shot examples in the QOC format: accuracy rose only 3%, the gap remained. Rejected.

  2. Lost-in-the-middle? Maybe the model forgets the options after reading a long context in QOC? They checked: the model remembers the options perfectly. Rejected.

  3. Causal attention? The key experiment compared three architectures:

    • Decoder-only (causal mask) → gap 14.7%
    • Encoder-decoder (bidirectional encoder) → gap 2.3%
    • Encoder-only (fully bidirectional) → gap 0.02%

    Bingo! Only models with a causal mask show the problem. Encoder models, where every token sees every token, don't suffer.

The final proof: The researchers ran three interventions on the model:

  • Attention pruning in CQO: They manually blocked attention from the options to the context (simulating QOC). CQO accuracy dropped by 26.8%. Confirms that the options' access to the context is critical.

  • Activation patching in QOC: They took the internal option representations from a CQO run and patched them into a QOC run. QOC accuracy rose by 6%. Confirms that the problem is precisely that option representations form without the context.

  • QOCO (repeating the options): A simple prompt hack with no model surgery. Accuracy rose by 8.2%.

All three interventions worked as predicted, which proves the cause isn't behavioral (what the model does) but structural (what the model CAN do given its architecture).

08

Original from the Paper

The researchers used a standard format for decoder-only models:

CQO Template:

{context}

Question: {question}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}

Answer:

QOC Template:

Question: {question}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}

{context}

Answer:

QOCO Template (workaround):

Question: {question}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}

{context}

Choose from: A | B | C | D

Answer:

Evaluation compared the log-probabilities of all options at the "Answer:" token position. The option with the highest probability was taken as the model's prediction.

09

Adaptations and Extrapolations

💡 Adaptation for comparative analysis:

The principle works not only for classic MCQA but for any task where you need to choose between options based on criteria.

Task: Pick the positioning for a new product

**Business context:**
SaaS for HR teams at small businesses (10-50 people). Current customers: 
manufacturers, retail, services. Customer pain points: "Excel can't keep up", 
"no time for paperwork", "we keep forgetting about health insurance and sick leave". 
Competitors: expensive all-in-one suites (Workday) or narrow tools (time tracking).

**Question:**
Which positioning will land best?

**Options:**
A. "Enterprise-grade HR, built for small companies"
B. "Ditch the busywork: sick days, vacations, paperwork on autopilot"
C. "Digital HR: one system instead of ten spreadsheets and chats"
D. "Save your HR manager 15 hours a week"

Pick a letter and explain it through the audience context.

💡 Adaptation for evaluating ideas:

**Situation context:**
The audience: e-commerce marketers (performance specialists, 25-35, 
New York/San Francisco). Pain points: reports across 5 platforms, hard to show ROI to the director, no time 
for strategy. Willing to pay $150-300/month for a tool that solves the problem.

**Question:**
Which product idea best fits this audience?

**Options:**
A. An AI assistant that optimizes ad campaigns in real time
B. One dashboard for all ad platforms + automated reports for leadership
C. A library of test hypotheses + results tracking
D. A performance-marketing learning platform with certification

Pick a letter and explain why through the audience's pain points and context.

🔧 Technique: varying the context → different decisions

Change the context while keeping the question and options, and you'll see how the context drives the choice:

**Context #1:** Startup, seed round, 12 months of runway, focus on growth
**Context #2:** Profitable business, 5 years on the market, focus on customer retention

**Question:** Where should we invest the marketing budget?
**Options:**
A. Content marketing (blog, SEO)
B. Performance ads (paid social, search ads)
C. Email marketing to existing customers
D. Affiliate program and referral system

Pick a letter for context #1, then for context #2. Explain the difference.

The model will give different answers depending on the context, which shows how well it accounts for the criteria.

💡 Adaptation for working with documents:

When you need to find an answer in a long document (a manual, a contract, a spec):

**Document:**
{long_contract_or_manual_text}

**Question:**
{specific_question_about_the_document}

**Options:**
A. {possible_answer_1}
B. {possible_answer_2}
C. {possible_answer_3}
D. {possible_answer_4}

Pick an option and quote a passage from the document as evidence.

The context (the document) comes first, so the model can match the options against specific facts in the text.

10

Resources

Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Authors: Hyunjong Ok, Jaeho Lee
POSTECH, HJ AILAB

The study builds on work on prompt sensitivity (Lu et al., 2022, Fantastically ordered prompts), the lost-in-the-middle effect (Liu et al., 2024), attention analysis in transformers (Clark et al., 2019; Abnar & Zuidema, 2020), and causal interventions and activation patching (Meng et al., 2022, Locating and editing factual associations in GPT).

Research digest

Key result

Shuffle your prompt elements and lose 15% accuracy. The cause isn't the model's training, it's the architecture. The CQO format (context→question→options) gives 15% more accurate answers on multiple-choice tasks than QOC (question→options→context). The trick: decoder models (GPT, Claude, LLaMA) use a one-way attention mask. Each token sees only what came BEFORE it, never what comes AFTER. Put the options BEFORE the context and they physically can't use the information in it. The model picks blind.

Principle

Decoder-only models work like a one-way street: a token sees only the tokens before it. In the QOC format, the options come before the context. When the model processes the options, the context doesn't exist for them yet: the attention mask blocks access to later tokens. The internal representations of the options form WITHOUT any information from the context. Even if the final answer token can see the whole text, it's too late: the option representations are locked in. In CQO the context comes first. When the model reaches the options, they see the entire context through attention. The comparison is made against evidence, not by guessing.

Why it works

An experiment measuring each element's contribution to the answer showed: in CQO the context accounts for 79.7% of the contribution to the prediction, in QOC only 33.5%. What's more, QOC accuracy = the accuracy of the QO format (no context at all). The context is structurally out of reach because of the mask, so the model picks blind based on general patterns. The gap grows with context length: a short context (~70 words) has a 6% gap, a long one (~305 words) 21%. The correct answer's position matters too: option D (closer to the context in QOC) suffers less (gap 10%), option A (farther away) more (gap 22%). The QOCO workaround works because the repeated options come AFTER the context: they see it through the mask and form context-aware representations. That lifts QOC accuracy by +8.2%.

When to apply

Multiple-choice tasks → whenever you need to choose among options based on context (requirements, criteria, facts), especially if the context is long (200+ words) or the correct answer could be in any position on the list. NOT relevant for encoder-only (BERT) and encoder-decoder (T5) models: they have bidirectional attention, so every token sees every token.

Mini recipe

1. CQO structure: context first (facts, requirements, criteria), then the question, and the answer options with letter labels A/B/C/D at the end 2. If the structure strictly requires QOC: use the QOCO workaround and repeat the options AFTER the context. The second time can be short: `Choose from: A. React | B. WordPress | C. Laravel | D. Squarespace` 3. For long contexts (200+ words): the difference is especially critical (up to 21%), so stick strictly to CQO. For short ones (under 100 words) the effect is weaker (6-7%), but it's still there

Examples

[BAD]: `Which technology fits the project? A. React+headless CMS B. WordPress+WooCommerce C. Laravel+Vue D. Squarespace. Project: furniture online store, priorities: fast mobile loading, a simple admin panel for non-technical staff, QuickBooks integration, budget $8,000, timeline 2 months` [GOOD]: `Context: Online store for a furniture company. Priorities: fast loading (customers on mobile), a simple admin panel for non-technical managers, QuickBooks integration. Budget $8,000, timeline 2 months. Stability matters more than creativity. Question: Which proposal best fits the requirements? Options: A. React+headless CMS, custom design, $10,000, 3 months | B. WordPress+WooCommerce, ready-made theme, $7,000, 1.5 months | C. Laravel+Vue, bespoke architecture, $9,000, 2.5 months | D. Squarespace+external cart, visual editor, $5,500, 1 month. Answer with the letter and explain via the priorities`

In plain words

Here's the deal: modern AI models aren't an all-seeing eye, they're hostages to their own architecture. Models like GPT or LLaMA use a causal mask that forces them to read text strictly left to right. The model literally can't see what's written "ahead" until it gets there. That's why word order in your request decides everything: if you ask the question first and only then hand over the data, the model tries to work out the point without having the context in memory yet. It's not just an inconvenience, it's a fundamental barrier that breaks the computation at the deepest level.

It's like assembling an IKEA wardrobe when the instructions are printed on the bottom of the box and you've already dumped every part on the floor. You stare at a pile of screws and try to guess what they're for, instead of reading the plan first and then picking up the tools. You either get it wrong or burn a ton of energy redoing it. When you put the question at the end, the model works like a pro with the blueprint in hand; when you put it at the start, it's a confused newbie fumbling in the dark.

The study showed that the CQO format (context → question → answer options) is the gold standard. It's 14.7% more accurate than asking the question first and dumping the data afterward. That gap is huge: it's the difference between a "smart assistant" and a "buggy bot". And every model has this ailment: GPT-4, Claude, LLaMA, Qwen and Gemma all slip up the same way. If you first ask it to pick the best option and then unload three pages of descriptions, the model simply loses focus and guesses, because its architecture doesn't let it properly "digest" the question in advance.

The principle is universal and goes well beyond tests. Whether you're choosing a vendor from five proposals, analyzing a legal contract or asking for a call summary, always serve the meat (the context) first. If you open with "Which of these candidates is best?" and then attach the resumes, you're sabotaging the answer's accuracy yourself. SEO for prompts now looks like this: facts first, then the task, and only at the very end the answer format. Any other order is shooting yourself in the foot.

Bottom line: stop hoping the model will "figure out" your stream of consciousness on its own. It's a calculator that processes tokens one by one, and if you feed it data in the wrong order, it just spits out garbage. Want a sane result? Use the CQO layout. It's the simplest, free way to instantly boost any model by 15% just by reordering your paragraphs. Ignore the architecture and you get hallucinations instead of analysis.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on CQO vs QOC: Why the Order of Prompt Elements Swings Accuracy by 15% arXiv:2601.14152

Sign in to apply this study New accounts get 50 free credits.

Related techniques

Instruction Order and Placement