TL;DR
System-2 Counting is a technique for counting large lists with an LLM by splitting them into small chunks. Instead of counting 50 items at once (where the model loses track), you split the list into chunks of 5-10 items with a | separator; the model counts each chunk separately, then adds them up. One request, but step-by-step work inside.
LLMs count layer by layer — the counter ticks up at each transformer layer. Up to 10 items the model confidently gives the right answer. Past 15-20 it starts making mistakes: 15 instead of 16, 20 instead of 21. Past 30-40, accuracy drops almost to zero. The cause isn't that the model doesn't know numbers — it's an architectural threshold: there aren't enough layers to accumulate a large counter. A model with 28 layers physically can't count accurately to 50 using its internal mechanism.
The fix mirrors a human cognitive strategy: break the hard thing into easy things. The | separator creates boundaries between chunks. The model counts within each chunk (fast System-1), writes the results out explicitly ("part 1: 4, part 2: 6"), then adds them up through reasoning (slow System-2). Accuracy jumps from 5% to 95% on lists of 50+ items.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
ONE PROMPT — THREE STAGES INSIDE:
Input data:
item, item, item | item, item | item, item, item, item
STAGE 1 (implicit): In each chunk the model counts with its internal mechanism
→ Counters: 3, 2, 4
STAGE 2 (explicit): The model outputs intermediate results
→ Text: "part 1: 3, part 2: 2, part 3: 4"
STAGE 3 (explicit): The model adds them up
→ Final answer: "Total: 9"
Important: Each chunk should be within the model's reliable counting range (usually 5-10 items). If the model counts accurately up to 15, use chunks of 10-12. If only up to 8, use 5-7.
Example
Task: You have the transcript of a two-hour startup podcast. You need to count how many times the host used "like" as a filler word (not in the sense of "similar to" or "enjoy").
A classic prompt like "count how many times 'like' appears" will give an inaccurate result on a long text. The LLM will lose track after 20-30 mentions.
Prompt:
Here is a podcast transcript, split into parts with the | symbol.
Count how many times the word "like" is used as a filler word in each part separately.
Then add up all the parts.
Answer format:
Part 1: [number]
Part 2: [number]
...
Total: [sum]
Text:
[5 minutes of transcript] | [5 minutes of transcript] | [5 minutes of transcript] | ...
Result:
The model will give a step-by-step count for each block of the transcript, then the total. Instead of a vague "about 40-50" you'll get an exact "47". You'll see the breakdown by part — which lets you check the logic and spot it if the model badly miscounted in some part.
Why It Works
An LLM isn't a calculator with variables; it's a pattern-based text generator. When the model counts, it doesn't keep a variable like count = 17 in memory. Instead, information about the quantity is smeared across the transformer layers — at each layer the activations shift a little, encoding "how many so far". It's similar to how a person senses "a lot" or "a little" without naming an exact figure.
Up to 10 items, this internal mechanism works cleanly. The model is trained on patterns of short lists and accumulates the information precisely. Past 15-20, compression kicks in: the model encodes numbers logarithmically rather than linearly (just as human intuition clearly tells 2 apples from 3, but 47 from 52 only vaguely). Past 30-40, the transformer layers saturate — they simply don't have the depth to encode such a large number through layer-by-layer changes.
System-2 Counting uses the LLM's strength — reasoning through text. The | separator creates explicit boundaries. The model handles each chunk within its reliable range (up to 10), writes out intermediate results as text ("part 1: 4"), and then works with those numbers like any other text in the task. The final sum is no longer an internal counter but arithmetic through reasoning. The model is great at "4 + 2 + 3 = ?" — it's a simple pattern task.
Levers you control:
Chunk size — shrink it for weaker models (5-7), grow it for stronger ones (12-15). Test a few lists to see where your model starts making mistakes.
Output instruction — "no explanations" gives just the numbers; drop that phrase if you want to see how the model reasons ("in the first part there are three apples and one pear...").
Boundary format —
|is a clear separator, but you can use===or---if the text already contains vertical bars.
Prompt Template
Count {what_to_count} in the following list, split into parts with the | symbol.
Steps:
1. Count within each part separately
2. Write down the result for each part
3. Add up all the parts
Answer format:
Part 1: [number]
Part 2: [number]
...
Part N: [number]
Total: [sum]
Data:
{item_1}, {item_2}, ... | {item_N}, {item_N+1}, ... | ...
How to fill it in:
- {what_to_count} — exactly what to count ("the word 'like'", "brand mentions", "all items in the list")
- Split the data into parts with | so that each part has 5-10 items (the reliable range for most models)
🚀 Quick start — paste into the chat:
Here's a System-2 Counting template for accurate counting in long lists.
Adapt it to my task: [your counting task].
Ask me:
- What exactly needs to be counted?
- What chunk size is optimal for my model?
- How should my data be split?
[paste the template above]
The LLM will ask about your data and the reliable counting range — it needs that to split things correctly. It will take the structure from the template and build a prompt for your task.
Limitations
⚠️ Data needs pre-processing: You have to split the list into chunks manually before feeding it to the model. If you have a 1000-row CSV, you'll first need to insert separators every 10 rows. Automating this takes a script or an intermediate step.
⚠️ Not for every kind of counting: The method works for explicit lists of items. If you need to count something implicit (say, "how many times a negative emotion appears in the text"), the model still has to analyze the whole text at once, and chunking can break the context.
⚠️ The threshold depends on the model: Gemma 27B (62 layers) counts reliably up to 20-25 items; Llama 3 8B (32 layers) — up to 10-12. You need to know your model's capabilities or test it on short examples.
How It Was Studied
A team from Sharif University of Technology started with a simple observation: models count confidently up to 10, then accuracy drops. They tested three open models (Qwen2.5 7B with 28 layers, Llama 3 8B with 32 layers, Gemma 3 27B with 62 layers) and two proprietary ones (GPT-4o, Gemini-2.5-Pro) on lists of 1 to 100 items.
They measured two things: accuracy (exact match) and mean absolute error. For each list-size range (11-20, 21-30, 31-40 and so on) they ran 100 examples. The finding: Qwen2.5's accuracy on 31-40 items without structuring is 6%, with System-2 — 38%. On 11-20 items: 38% vs 95%.
They compared four strategies:
1. Unstructured list + short answer (just the number) — the System-1 baseline
2. Unstructured list + Chain-of-Thought (reasoning before the answer) — an attempt at System-2 without help
3. Structured list with | + short answer — the chunks are there, but the model doesn't use explicit steps
4. Structured list with | + CoT — full System-2 Counting
The result was surprising: without structuring, even CoT doesn't help. Llama 3 8B on 31-40 items with CoT reaches only 2% accuracy. But add separators — and accuracy jumps to 38%. For GPT-4o on 91-100 items: 11% without structure vs 86% with it.
Then they went deeper — mechanistic analysis. Using attention analysis and causal mediation (knocking out specific layers or attention heads and watching what breaks), they worked out exactly how information flows inside the model:
- Within each chunk, the list is processed by the usual System-1 mechanism — the counter accumulates across layers
- The final count information is localized in the chunk's last token (before the
|or after the last item) - Special attention heads in the middle layers carry this information over to the tokens that output the intermediate results ("part 1:", "part 2:")
- The final layers aggregate the intermediate numbers through ordinary arithmetic reasoning
This explains why it works: the model doesn't invent a new counting mechanism; it combines two known ones — the internal counter (System-1) for small chunks and reasoning through text (System-2) for aggregation. The | separator creates explicit checkpoints where the counter resets and starts over.
Adaptations and Extrapolations
🔧 Technique: Transparent counting → see where the model went wrong
In the original method the model outputs only the per-chunk numbers. Drop "no explanations" and ask for a detailed answer, and you'll see HOW the model counted:
Answer format:
Part 1: [list the items and count them] — [number]
Part 2: [list the items and count them] — [number]
...
This is useful when the count isn't obvious (you're counting only the items that match a criterion, not all of them). You'll see which items the model included and can correct it if it misinterpreted something.
🔧 Technique: Dynamic chunk size → adapt to complexity
In the original, all chunks are the same size. But if items vary in complexity, you can vary the size:
Simple items (words): chunks of 15
Complex items (paragraphs of text to analyze): chunks of 3-5
To count mentions in short tweets, you can use chunks of 20 tweets. To count arguments in long essays — 2-3 essays per chunk.
🔧 Technique: Hierarchical counting → for very large lists
If you need to count 500 items, use two levels of splitting:
LEVEL 1: Split the 500 into 10 blocks of 50
LEVEL 2: Split each block into 5 parts of 10
First count within each part of 10:
Block 1, part 1: 10
Block 1, part 2: 9
...
Block 1, total: 47
Then add up the blocks:
Block 1: 47
Block 2: 51
...
Total: 489
This is "divide and conquer" — for extra-long lists where even an intermediate sum of 50 numbers could go wrong.
Resources
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy The study builds on work on the internal counting mechanism in LLMs (Hasani et al., 2025), logarithmic number encoding (AlquBoj et al., 2025; Dehaene, 2011), and System-1/System-2 cognition theory (Kahneman, 2011)
Authors: Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah Sharif University of Technology