Prompt and output compression: fewer tokens without losing accuracy

What prompt compression is: input vs output

Prompt compression means getting the same result with fewer tokens. It splits into two different jobs that are often confused:

• Input compression: shrinking what you send. That covers rewriting instructions in a denser notation, extracting the facts from a long problem before solving it, summarising a document into a reusable digest, and pruning rules or glossary entries that do not apply to the case.
• Output compression: shrinking what the model writes back, including hidden reasoning. That covers brevity reminders, explicit answer formats, word limits and removing phrases that invite extra deliberation.

The distinction matters for cost. In the CAVEWOMAN study, output tokens cost 4-8x more than input tokens, and the two directions behaved oppositely: telling the model to answer briefly cut total cost 1.4-2.4x, while telegraphing the prompt itself made things worse. The rest of this page explains which input cuts are safe, which output cuts work, and where the research says more context, not less, is the right move.

Why it works

Three mechanisms recur across the studies.

Filler competes with content. Models narrate. The Focused Chain-of-Thought paper estimates that up to 50% of generated tokens are filler such as restating the conditions and "let's break this down". When the facts were extracted into numbered tags first and the model solved from those alone, Qwen-3 models used 2-3x fewer tokens at comparable accuracy, and an overthinking score fell from 2.35 to 1.74. The model had nothing left to re-read, so it went straight to computing.

Rules beat judgment calls. Asking a model to "shorten" makes it decide what matters, and it tends to drop small numbers, qualifiers and causal links. Telegraph English reframes compression as re-encoding under a strict grammar: one statement per line, numbers always with units, symbols for cause and conclusion. On 4,081 fragments from LongBench-v2 it used 50% fewer tokens while keeping 99.1% accuracy on key facts.

Verbosity is a default, not a requirement. The BenchOverflow study found that ordinary words like "all", "every" and "in detail" push models toward the token limit, and that a one-line reminder ("provide a concise, precise response without unnecessary elaboration") reduced cap-hitting answers across 9 models by 20-60% without hurting answer quality on regular tasks.

Variants that tested well

1. Symbolic notation for logic. MetaGlyph replaces wordy filters with operators the model already knows, e.g. select where ∈(e-commerce) ∩ ¬(enterprise), for 62-81% token compression. Kimi K2 reached 100% accuracy on selection versus 90.8% with prose. Operator support is uneven: the intersection operator scored only 21.4% even on GPT-5.2, and mid-size 7-12B models showed 0% symbol understanding.
2. Extract, then reason. Beyond F-CoT, a compressor-predictor split summarises a long document by criteria first and answers from the summary; a larger compressor wrote 4.6x fewer tokens yet was 60% more accurate. Chain of Summaries refines a summary with self-generated questions; a 170-token summary scored 0.80 F1 versus 0.76 for the 11,000-token document. BRAID swaps prose reasoning for a Mermaid graph.
3. Prune what does not apply. WARPP removes rule branches irrelevant to the current case before the model sees them. SLoW gives glossary hints only for the 5-10 rarest words, because a full glossary adds noise. A five-block 5C prompt used 54 input tokens where an XML-heavy version used 348, with similar output. The semantic-density paper argues for cutting polite filler, but its reported gain was not verified by the authors' own table, so treat it as a principle rather than a measurement.
4. Bound the output. Put a brevity line and an answer template at the top, ideally in the system prompt. AdvPrompt found persuasive brevity frames compressed reasoning answers by 63-78% on simple tasks, while threat-based frames managed only 4-25%; note that its best frame cites an invented study. For coding agents, the bounded-efficiency study found "compare several approaches" inflated reasoning tokens 2.4-7.4x with no change in the code, and replaced it with scope, a done criterion and a stop condition.

When shorter costs more

Do not compress the request itself. CAVEWOMAN tested the popular "write like a caveman" tip on eight models and 11,465 tasks. Cutting the prompt by 15% raised total cost by 3-15%, up to 2.7x in the worst cases, and lowered accuracy: the model compensated for the missing context with longer answers.

More context can be the fix. Two counter-studies argue against brevity as a goal:

• The ACE playbook study watched a full rewrite collapse an 18,282-token context to 122 tokens, with accuracy falling from 66.7% to 57.1%. Growing a detailed playbook in small additions instead beat baselines by 10.6% on agent tasks and 8.6% on finance.
• The DeepNews study found that long analytical articles written from under 15,000 characters of context were hallucination-free only 40% of the time, rising to 85% at 30,000+ characters, roughly ten times the output length.

Other limits:

• Re-encoding can still drop nuance: in the Telegraph English paper "30 calendar days" lost the word "calendar", and short dense texts such as tables grew rather than shrank.
• A brief answer may take a different path. CAVEWOMAN reports that in roughly half the cases a correct brief answer did not match the unconstrained answer in meaning, and hidden reasoning in models such as GPT-5.4 was not shortened by the instruction.
• Hard problems compress less. AdvPrompt saw about 20% compression on competition math versus much more on easy tasks; F-CoT gave no gain on already compact problems.

Rule of thumb: keep the request complete, prune what is irrelevant, compress the answer, and never trade away the facts the task depends on. For deciding when to switch reasoning off entirely, see the adaptive reasoning page.

Reminder: Please provide a concise, precise response without unnecessary elaboration.

{your_task}

Answer {brevity_style}.
Remove: introductory phrases, repetition, "it's worth noting," explanations of the obvious.
Keep only: {what_to_keep}.
Answer format: {structure_template}.
{exception_if_needed}

{your_usual_request_unchanged}

Use ONLY the facts from <context> below. Do not refer to the original problem.

Instructions:
1) Reason step by step
2) When you use a fact, cite <info_k>
3) Final answer in \boxed{}

<context>
<info_1>Fact 1</info_1>
<info_2>Fact 2</info_2>
<question>What needs to be found</question>
</context>