Selective Reasoning: When to Turn On Chain of Thought in an LLM

STEP 1: Classify the request → determine intent (knowledge vs reasoning)
STEP 2: Routing → choose a strategy (direct answer vs CoT)
STEP 3: Inference → execute with the chosen strategy

In infrastructure, every step is automatic. In a chat, you implement it manually via a meta-prompt or by splitting work across chats.

Task: A product manager at an Austin EdTech startup is evaluating a new feature — adaptive assignments for students. There's a mix of simple questions ("Which engagement metrics should we track?") and complex ones ("How do we measure the impact of adaptivity on retention, segmented by knowledge level?").

Prompt:

You are an assistant to a product manager in EdTech. Before answering a question:

1. Determine the task's complexity:
   - SIMPLE: a fact, a definition, a list of metrics, well-known information
   - COMPLEX: math, cause-and-effect, trade-offs, multi-step analysis

2. Choose the response mode:
   - For SIMPLE → a direct answer with no reasoning
   - For COMPLEX → chain of thought: break the problem into parts, analyze step by step, draw a conclusion

Question: How do we measure the impact of adaptive assignments on student retention, segmented by knowledge level?

Result: For the simple metrics question, the model will return a list (completion rate, time on task, progress speed) without unnecessary reasoning — fast and cheap. For the complex retention question, you get a step-by-step analysis: breaking retention down by segment (beginners/advanced), estimating the impact of adaptivity on each segment, calculating the net effect, accounting for confounding factors, and a final conclusion. Token savings on the simple requests, depth on the complex ones.

LLM weakness: By default, a model either always reasons (slow, expensive) or never does (losing accuracy on complex tasks). An LLM can't decide on its own when it needs depth and when it needs speed. The result: overpaying in tokens on simple questions or poor accuracy on complex ones.

LLM strength: Models are excellent at classifying intent and can follow conditional logic in a prompt. An LLM understands the difference between "the capital of France" (knowledge) and "optimizing a portfolio given correlations" (multi-step analysis). No fine-tuning required — just a clear instruction.

The workaround: By adding an explicit condition to the prompt — "if the task is complex, use CoT; if it's simple, answer directly" — we give the model a lever to pick its own strategy. The model classifies the request (its strength) and applies the right mode. It's like letting a driver choose between the highway and a back road depending on the destination.

Control levers:

• Classification criteria: Spell out what counts as "complex" — add concrete patterns for your field to the prompt ("requires calculations," "many variables," "non-obvious relationships")
• CoT depth: Specify the reasoning format — "a brief 3-step analysis" to save tokens or "a deep breakdown with justifications" for quality
• Fallback strategy: Add "if you're unsure about the classification, use reasoning" as a safety net for edge cases

You are a {role, e.g.: data analyst, developer assistant, marketer}.

Before answering a question:

1. Determine the task's complexity:
   - SIMPLE: a fact, a definition, a list, well-known information, a standard process
   - COMPLEX: math, logic, cause-and-effect, trade-offs, multi-step analysis, non-obvious patterns

2. Choose the response mode:
   - For SIMPLE → give a direct answer with no reasoning
   - For COMPLEX → use chain of thought:
     a) break the problem into parts
     b) analyze each part
     c) synthesize a conclusion

Question: {user_question}

What to fill in:

• {role} — the context the model works in (analyst, developer, consultant, copywriter)
• {user_question} — your request

Boosting the prompt: Add concrete classification examples for your field. For marketing, say: "SIMPLE — picking social media channels, COMPLEX — conversion attribution in a multi-touch funnel."

⚠️ Manual overhead: In a chat, classification happens on every request — the model spends a few tokens figuring out the complexity. For one-off questions that's overhead. In the infrastructure solution (Semantic Router), classification lives in a separate fast module.

⚠️ A blurry line: Medium-difficulty tasks (for example, "Explain why rising inflation affects tech stocks") can be classified either way. The model may pick the wrong mode — using CoT where it isn't needed, or vice versa.

⚠️ No automation: Unlike Semantic Router (optimization built in at the infrastructure level), in a chat the user steers the logic through the prompt by hand. It works, but it takes awareness and tuning to your own tasks.

The IBM team took 12,000 queries from the MMLU-Pro benchmark — 14 knowledge domains from math to history — and ran them through several modes: (1) always CoT, (2) never CoT, (3) Semantic Router with classification. They measured three metrics: answer accuracy, latency (time to answer) and token count.

The results were surprising: on knowledge tasks (history, business, economics), reasoning not only didn't help but sometimes hurt — the model overthought a simple question and got it wrong. Accuracy dropped by 5-10%. On math and physics, though, CoT delivered +20-30% accuracy — without a step-by-step breakdown the model couldn't cope.

The main insight: most requests (60-70%) don't need reasoning, and on those we overpay up to 150x in energy and nearly double in tokens. Applying CoT selectively saved 48% of tokens and cut latency in half, while overall accuracy rose by 10% — because the model stopped getting simple questions wrong by overthinking them.

💡 Adaptation for personal chats:

Set up two permanent chats in Claude/ChatGPT for different kinds of tasks:

Chat 1: "Quick Answers"

System prompt: Answer briefly and to the point, with no extra reasoning.
You are a knowledge base: facts, definitions, lists, standard processes.

Chat 2: "Deep Analysis"

System prompt: Use chain of thought for every answer.
Break the problem into parts, analyze step by step, justify your conclusions.

Route tasks between the chats by hand. You save tokens on your subscription and get speed on simple questions.

🔧 Technique: Two-step request → explicit classification

Instead of automatic classification inside one prompt, split it into two requests for more precision:

Request 1 (classification):

Determine the task's complexity using these criteria:
- SIMPLE: a fact, a definition, a list, well-known information
- COMPLEX: requires calculations, causal analysis, trade-offs, multi-step logic

Task: {your task}

Answer with one word: SIMPLE or COMPLEX

Request 2a (if SIMPLE):

Answer the question briefly: {your task}

Request 2b (if COMPLEX):

Analyze the task step by step using chain of thought: {your task}

Downside: Two requests instead of one — overhead.
Upside: Fully transparent classification, fewer errors, and you can reuse the classification result for similar tasks.

When to Reason: Semantic Router for vLLM
Chen Wang, Yue Zhu, Xunzhuo Liu, Xiangxi Mo, Yuhan Liu, Junchen Jiang, Huamin Chen
IBM Research, UC Berkeley, University of Chicago, Red Hat
NeurIPS 2025 Workshop: MLForSys
Open-source: https://vllm-semantic-router.netlify.app/

Related work:

• FrugalGPT (Chen, Zaharia, Zou, 2023) — cascade routing between models
• RouteLLM (Ong et al., 2025) — model selection based on preference data
• Chain-of-Thought Prompting (Wei et al., 2022) — the original CoT paper