TL;DR
Compressor-Predictor is a two-model architecture: the first compresses a long context into a short summary, the second answers based on the compressed version. The study shows that compression quality matters more than the size of the final model. A bigger compressor model isn't just more accurate — it's also shorter. Paradoxically, a 7B model generates 4.6× fewer tokens than a 1.5B one while being 60% more accurate. The mutual information between the context and the compression predicts final quality regardless of the task.
The main finding: scaling the compressor is 5× more effective than scaling the predictor. Growing the compressor from 1.5B to 7B improves accuracy by 60%, while growing the predictor from 70B to 405B gives only 12%. Meanwhile compute costs grow sublinearly — going from 1.5B to 7B in Qwen-2.5 adds only 1.3% FLOPs. The explanation: bigger models pack information more densely — each token of the compression carries more bits of useful information (bit efficiency is 5.5× higher).
The gist of the method: front-load compute into the compressor. Instead of "one request to a big model" → do "high-quality compression with a small model + an answer from a mid-sized model." You can run a powerful compressor locally (up to 27B fits on a MacBook without quantization) and the predictor in the cloud at lower cost. In practice, a 3B compressor + a mid-sized predictor deliver 99% of a frontier model's accuracy at 26% of the API cost.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
CONTEXT (X) → a long document, article, transcript
QUERY (Q) → a question or task
STEP 1: Compressor
Input: Context X + Query Q
Output: Compressed summary Z (short but informative)
STEP 2: Predictor
Input: Compressed summary Z + Query Q
Output: Final answer Y
Important: The steps run as separate requests. You can use different models or even different services.
Example
Task: You're reading a long Medium post about a new HR-tech SaaS startup (10,000 words: problem, solution, team, financials, metrics). You need an expert assessment — is it worth investing? But if you feed the whole post into one request, the model will spread its attention across the details and miss what's critical.
Request 1 (Compressor):
Read this post about a startup. Pull out only the facts critical to an investment decision:
- Problem and solution (1-2 sentences)
- Key metrics (revenue, growth, retention)
- Team (founders' experience)
- Risks (what could break)
8 sentences max. No fluff, just facts with numbers.
[paste the long post]
Request 2 (Predictor):
You are a partner at a venture capital fund. Here's a short summary of a startup:
[paste the summary from the first request]
Give your assessment: invest or pass? Justify it in terms of: market size, product-market fit, team, unit economics. Format: verdict + 3 key arguments.
Result:
The first request returns a dense summary: "Startup X solves problem Y with Z. Revenue $1.2M, 300% YoY growth, 85% retention. Founders are ex-Google with 10 years of HR experience. Risks: narrow niche (IT companies only), dependence on a single channel (LinkedIn ads)."
The second request produces a structured investor assessment with specific arguments, based on the compressed facts. The answer will be more precise than if you fed in the whole post at once — because the model works with distilled information instead of trying to hold 10k words.
Why It Works
LLMs suffer from context rot — accuracy drops as the context grows. The model "spreads" its attention across details, misses what's critical, and produces generic phrases. It hurts most at 10k+ tokens: the model technically fits it all, but quality degrades.
LLMs are excellent at explicit instructions on short input. Give a model 200 tokens of dense summary with a specific task and it will give a precise answer. That's its strength: structured input → clear output.
The two-step process uses this strength twice. Step one: "pull out only what matters by criteria X, Y, Z" — the model filters out noise and keeps the signal. Step two: the predictor works with a clean signal without spending its "attention budget" hunting for a needle in a haystack. Compression lowers the cognitive load on the predictor.
Why are bigger compressors shorter? The paradox is explained by information density. A small model (1.5B) isn't sure what matters — it hedges and includes extra stuff. A big model (7B) is better at telling signal from noise — it packs facts more densely, and each token carries more bits of useful information. Result: a 7B compressor generates 4.6× fewer tokens with 60% higher accuracy.
Control knobs:
Compression length — specify "N sentences max" in the first request. Smaller N → faster, but you risk losing details. Larger N → slower, but safer for complex tasks.
Compression criteria — explicitly list WHAT to extract ("metrics," "risks," "team"). Without criteria, the compressor decides what matters on its own — and may get it wrong.
Compressor model — if a more powerful model is available (or one running locally on your device), use it for the first step. Compression quality matters more than predictor speed.
Output format — add "in the format: point 1, point 2..." — a structured summary is easier for the second step to use.
Prompt Template
Request 1 (Compressor):
Read the {content_type}: {context}
Task: {task_or_question}
Extract only the information critical to this task. Write a summary covering:
- {criterion_1}
- {criterion_2}
- {criterion_3}
{N} sentences max. Facts only, no interpretation.
Request 2 (Predictor):
You are {expert_role}. Here's a short summary:
{paste_summary_from_request_1}
Task: {task_or_question}
{answer_format_instructions}
Placeholder guide:
{content_type}— "article," "document," "podcast transcript," "technical documentation"{context}— the long text to process{task_or_question}— a specific question or goal ("assess investment potential," "extract instructions," "find risks"){criterion_1,2,3}— aspects that must make it into the summary ("key metrics," "technical requirements," "team"){N}— the compression length limit (5-15 sentences, depending on the task){expert_role}— context for the predictor ("investor," "technical architect," "lawyer"){answer_format_instructions}— how to structure the final answer ("verdict + 3 arguments," "step-by-step instructions," "pros/cons table")
🚀 Quick start — paste into the chat:
Here's a two-step prompt template for processing long context. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste both templates above]
The LLM will ask: "What context are we processing? What's the end goal? Which criteria matter for the summary?" — because it needs to understand WHAT to compress and BY WHICH criteria to filter. It will take the "compress then answer" pattern and adapt it to your task.
Why the Two-Step Process Is Effective
Experiments on 5 datasets (medicine, finance, scientific papers, chats, web pages) showed:
1. Scaling the compressor is 5× more effective than scaling the predictor
Growing the compressor from 1.5B to 7B → accuracy +60% (LongHealth), +160% (FinanceBench). Growing the predictor from 70B to 405B → accuracy +12% (LongHealth), +1% (FinanceBench). Compression quality determines the final result; predictor size is secondary past 8-70B.
2. Bigger compressors are shorter and denser
7-12B models generate 4.6× fewer tokens than 1-1.5B ones while keeping accuracy. The paradox: more parameters → shorter output. The explanation is bit efficiency (bits of mutual information per token): a 7B model packs 5.5× more bits per token than a 1.5B one. It's not just smarter — it encodes information more densely.
3. FLOPs grow sublinearly as the compressor scales
Qwen-2.5: going from 1.5B to 7B adds only 1.3% FLOPs-per-generation. Why? The model is 4.6× bigger but generates 4.6× fewer tokens — the effects cancel out. Bottom line: you can scale the compressor almost for free in compute terms and get +60% accuracy.
4. Mutual information predicts quality regardless of the task
The mutual information between the context and the compression (I(X; Z | Q)) correlates with accuracy (R² = 0.71) and perplexity (r = -0.84). It's a task-agnostic metric — you can assess a compressor's quality without running it on a specific task. Think of it as perplexity for compression.
5. Factor importance hierarchy: compressor family > compressor size > predictor size
Regression analysis showed that the choice of compressor model family (Qwen vs Llama vs Gemma) matters more than size. Qwen-2.5 consistently beats Llama at the same sizes. Next comes compressor size. Predictor size matters least.
6. Applicability to Deep Research
On the DeepResearch Bench (multi-step research tasks), a local 3B compressor + a cloud predictor deliver 99% of a frontier model's accuracy at 26% of the API cost. A real-world case: instead of GPT-4o at every stage → a local Qwen-3B for processing + GPT-4o for the final step.
Conclusion: The two-step architecture isn't just a "division of labor" — it's an economically optimal pattern. Front-loading compute into the compressor (which can run locally, for free) lightens the load on the expensive cloud predictor without losing quality.
How They Tested It
Datasets: 5 different corpora — LongHealth (synthetic medical histories, 20 patients, QA), FinanceBench (150 10-K financial reports, QA), QASPER (5049 scientific papers, QA), WildChat (conversations with LLMs), FineWeb (CommonCrawl web pages). Coverage: medicine, finance, science, chats, the open web.
Compressors: Llama-3 (1B, 3B, 8B), Qwen-2.5 (1.5B, 3B, 7B, 14B, 32B), Gemma-3 (2B, 9B, 27B). Open-source models, not reasoning architectures (GPT-style transformers). Prompt: "compress the context into N sentences, keeping only what's critical for the question."
Predictors: GPT-4o (baseline), Llama-3 (1B, 8B, 70B, 405B), Qwen-2.5 (72B). Prompt: "answer the question based on the summary."
Metrics: - Accuracy (QA tasks) — a GPT-4o-mini judge compares the answer with the ground truth - Perplexity (language modeling) — log probabilities under Llama-3.1-8B - Mutual Information — a Monte Carlo estimate of I(X; Z | Q) via log probabilities. A proxy model (7-8B from a different family) computes the probabilities to avoid the miscalibration of small models. - Bit efficiency — MI per token (how many bits of information each compression token carries) - FLOPs-per-generation — model parameters × number of tokens
Procedure: 5 random seeds per experiment. They varied: compressor family, compressor size, predictor size, compression length (3/6/9 sentences). They built rate-distortion curves (rate = MI per token, distortion = 1 - accuracy) and fit an exponential decay to characterize the trade-off.
Validation: Ablation with different proxy models for MI — the trends hold. Estimating MI without a proxy (under the compressor itself) — identical behavior. Multi-turn interactions — the patterns hold.
Compressor error analysis: 36.3% — wrong answer in the compression, 33.3% — no answer, 30.4% — missed details. This shows the compressor either makes factual mistakes or filters too aggressively.
Limitations
⚠️ Two requests instead of one: The method requires sequential requests — compressor first, then predictor. In API systems this is automated; in a chat, you copy by hand between requests. For simple tasks on short context, the overhead isn't worth it.
⚠️ Tokens spent on compression: The compressor generates intermediate text that eats tokens. Even though bigger compressors are shorter (4.6× fewer), total token use (context + compression + answer) is higher than a single request. The savings show up in quality, not token count.
⚠️ Depends on the quality of the criteria: If you don't say WHAT to extract in the first request, the compressor decides on its own. It may miss something critical. The method requires deliberately wording the compression criteria, which isn't always obvious up front.
⚠️ Not for every task: The two-step process helps on long contexts (5k+ tokens) with extractable information (facts, metrics, arguments). For creative work, subjective judgments, or tasks where "every nuance matters," compression hurts. For example: tone analysis, literary criticism, legal review — better to send one request to a big model.
⚠️ Risk of losing context: Aggressive compression (5 sentences out of 10k words) can throw out a critical detail. The compressor doesn't perfectly know the future question — it guesses "what matters." If the task needs a rare fact from the middle of the document, it may not make it into the summary.
Resources
An Information Theoretic Perspective on Agentic System Design — Shizhe He, Avanika Narayan, Ishan S. Khare, Scott W. Linderman, Christopher Ré, Dan Biderman (Stanford University, Wu Tsai Neurosciences Institute)