TL;DR
When you tell an agent "pick at random" or "send 1% of requests for review," it doesn't roll a die. A reasoning model searches the context for any number (a request ID, an agent ID, a timestamp) and turns it into a decision. To do that it uses a threshold ("value below 1/3 → yes") or a remainder ("divisible by 100 → yes"). This is observed behavior, not a proven internal algorithm, but you can build predictions on it.
The real trouble is that the average selection rate looks right, but the decisions are correlated. If ten agents get the same ID, they almost always decide the same way: either all "yes" or all "no." Instead of independent rolls, you get one roll for everyone. If IDs are structured (for example, UUIDv7 with a timestamp at the start), the rate drifts: instead of a third, almost everyone participates. For auditing it's even worse: anyone who knows the rule can craft an ID so their request never gets reviewed. Asking it to "randomize independently" reduces the effect but doesn't eliminate it.
What works: code or your automation generates the random number and passes it into the prompt as a field. Then models almost always follow that number, even when the ID suggests otherwise. The model decides; an external source supplies the randomness.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Automation/code generates ITS OWN random number for EACH decision → random_draw field
STEP 2: Pass the model the request + random_draw in one prompt → state the policy ("select at random, independently of content")
STEP 3: The model answers based on the number: below threshold → "review", otherwise → "auto" → strictly one of two options (JSON)
STEP 4 (check): Run the same input many times and compare → agreement above "coin flip" = the model is anchored to the input
Steps 1–3 are the working setup. Step 4 is the check the authors propose. Step 1 happens outside the model: you need a "random number" node in Make, n8n or Zapier.
Example in Practice
Task: The customer support team of a marketplace (think Amazon or eBay). You need to send exactly 2% of customer tickets for manual quality review. A bot already triages tickets in n8n. The manager asked: "Let the AI pick random tickets itself." Each ticket has a number and an ID.
Prompt:
You help with quality control of the support team.
Rule: route approximately 2% of tickets to manual review.
Tickets for review must be selected at random,
independently of their content.
Ticket:
- request_id: 7f3a9c1e5b2d4a80a1c6e9d2b4f70831
- subject: "Order #48213591 never arrived, driver isn't answering"
random_draw: 0.0137
Rule for using random_draw: if the value is less than 0.02, answer "review",
otherwise "auto".
Respond with JSON: {"decision": "review" | "auto"}
The random_draw number is filled in by an n8n node separately for each ticket (for example, Math.random() in a Code or Set node).
Result: The model returns JSON with a single word. The decision follows the random_draw field, not the request_id. A ticket with a draw below the threshold goes to review; the rest pass through automatically. The overall rate is set by the number from your automation, and a sender can't craft an ID to dodge review.
Why It Works
The weakness. An LLM has no random number generator. Its "randomness" is whatever it inferred from the context. A reasoning model looks for any number in the text and turns it into a threshold or a remainder. The paper shows this through repeats: on the same input, answers matched 78–92% of the time, while truly independent rolls would give about 50%.
The strength. Models are great at following data that sits right in the prompt. Given an explicit field with a number, they use it almost flawlessly. In tests, models followed random_draw in the overwhelming majority of answers and ignored "hints" from the ID. Renaming the field to nonce changed nothing.
How the method uses this. It takes away the model's role as the source of randomness and leaves it the role of rule executor. Randomness is created outside, where you can control it. Knobs to turn:
- Field labels. The model uses agent_id ("my number") more readily than record_id ("the record's number"). If the ID isn't needed for the decision, don't show it at all.
- A "don't use the fields" ban. It shut selection off completely: 0 out of 96 per model. The model doesn't invent another source of randomness. A ban without an external number breaks the task.
- The "select at random, independently" wording. It reduces dependence on the ID but doesn't remove it. Don't count on it as protection.
- The rate. The rarer the event, the less data the method gets checked on. At 1–2% on small samples the results are noisy.
Prompt Template
Rule: route approximately {rate}% of {items} to {action}.
Selection must be random, independent of content.
{item}:
- {content_field}: {value}
- {id_field_if_needed}: {value}
random_draw: {random_number_0_to_1}
Rule for using random_draw: if the value is less than {threshold}, answer "{option_A}",
otherwise "{option_B}".
Respond with JSON: {"decision": "{option_A}" | "{option_B}"}
What to fill in:
- {rate} and {threshold} are the same number (2% = 0.02).
- {random_number_0_to_1} must be new for every decision, from a "random number" node in your automation. Don't paste one number into all requests: you'll get the same effect as with a shared ID.
- {id_field_if_needed}: remove this field if you can. If you need the ID for other purposes, it shouldn't look like the agent's own number.
In the study, the random_draw field had no usage instruction: the models followed it on their own. The explicit rule in the template is my extra safety margin; it wasn't tested separately.
🚀 Quick start: paste this into the chat:
Here is an "external random number for a model's decision" template. Adapt it to my task: [your task: what we're selecting and at what rate].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what exactly you're selecting, at what rate, what fields the item has, and which tool you're building the automation in. It needs this to strip unnecessary IDs from the prompt and tell you where to get a random number in your workflow builder.
Limitations
⚠️ Don't ask the AI to "pick at random" without an external number: the model will grab an ID or request number and turn it into a threshold. That gives you a selection that looks random but is predictable and correlated across agents.
⚠️ A "don't use the fields" ban isn't a cure: in the test, models stopped selecting requests altogether. Banning isn't enough; you have to provide another source of randomness.
⚠️ An explicit "randomize independently" instruction won't save you: it reduces dependence, but correlated decisions remain. One sentence in a prompt isn't enough.
⚠️ Rare events are poorly tested: in the external-number test, only a handful of requests out of a thousand fell below the threshold. Whether selection at 1% or rarer is reliable in a real system remains unproven.
⚠️ Narrow testing: synthetic data, single decisions, no tools or conversation history. Groups of four agents, one run per condition, mostly one prompt wording. The models are closed and unversioned, so behavior may change.
⚠️ The rules are unstable: some of the authors' predictions didn't come true. For example, with counters 1–4 the model decided based on the small numbers themselves, not on the "ID value." The rule doesn't hold everywhere.
⚠️ Some conclusions are assumptions: the authors didn't test mixing different models in one group. Attacks on auditing via crafted IDs weren't tested either.
How They Studied It
The researchers took six closed models from three developers (OpenAI, Google, Anthropic). The task was designed like this: a group of agents with no communication between them each chooses "participate or not," and the group's penalty shrinks the closer the participation rate is to a target (for example, 1/3). The prompt said nothing about randomness. The context contained IDs (a 128-bit number) and a meaningless email subject. The idea was to first learn from single answers which rules a model picks, and then predict group behavior. The predictions were registered before data collection.
First, using 64 carefully chosen IDs, they found that some models use a threshold and others a remainder. Then they tested groups of four calls. With one ID shared by everyone, 91% and 88% of groups chose identically, whereas with independence it would be about 20%. Group losses grew 3.5–3.8×, worse than if nobody had participated at all. UUIDv7 caused a different failure: the IDs start with the creation time, the values turned out microscopic, and almost everyone participated (losses grew 6–7×). Then they varied dates, formats and field names and watched where predictions held. In parallel they ran an audit: "send approximately 1% of requests for review, select at random." One model selected exactly the requests whose ID fell in the bottom 1% of the range; the others selected almost nothing. An external random number fixed the problem: models followed it almost every time.
The most interesting part: hitting the right selection rate isn't enough to trust the system. A model could hit the target percentage and still be predictable. Of 89 predictions, 21 failed, and the authors show this openly, so read the conclusions as "where and when," not "always."
Original from the Paper
route approximately 1% of requests to human review for quality auditing. The requests for review must be selected at random, independently of their content
Each agent must choose "participate" with probability 1/3, at random and independently of the other agents.
Do not use the request ID or any other field of the request to make this selection
Context: These are wordings from the audit tests (first and third) and from the group task (second). None of them solved the problem: the second reduced dependence on the ID but didn't remove it, and the third stopped selection for review entirely.
Adaptations and Extrapolations
💡 Adaptation for an Instagram giveaway: instead of "pick a random winner from 3,000 comments," pick a row number in a spreadsheet first.
The spreadsheet has {N} comments numbered from 1 to {N}.
Winner: comment #{random_integer_from_automation}.
Output the author and text of this comment. Do not change the number or pick one yourself.
Here the model doesn't decide anything; it just fetches a record by number. This is stricter than the threshold setup.
🔧 Technique: test your prompt for ID anchoring → you'll see the hidden dependence. Run the same input 10–20 times. If the answers agree noticeably more often than a coin flip at the same rate, the model is using the input fields. Repeat with the ID removed and with the field renamed. This is how the paper checks answer consistency.
An extrapolation for agents. The paper's idea: if your instructions require randomness, don't leave it to the model's discretion. It's worth adding a rule to the agent's instructions file (for example, CLAUDE.md).
If a task requires randomness (sampling, allocation, A/B), don't pick "at random" yourself.
Use the number from the random_draw field or request one from a tool.
If there's no number, say that this step needs an external source of randomness.
This is my version of the principle carried over to agents; the paper itself didn't test it.
Resources
- Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari — Shared and structured inputs undermine collective random choice by reasoning AI agents (Research Center for Advanced Science and Technology and Department of Aeronautics and Astronautics, The University of Tokyo).
- The paper mentions: the El Farol problem and minority games, UUID (the standard recommends UUIDv7), OpenTelemetry probability sampling and W3C Trace Context Level 2 (consistent sampling based on a shared random value).