The Confident-Tone Effect: LLM Reviewers Inflate Scores for Self-Assured Wording

STEP 1: Rewrite the text in two versions (confident / cautious)
        under the rules "don't change facts, numbers, length, structure" → 2 versions
STEP 2: Check that the meaning survived (and the tone really changed)
        → manual comparison or a separate request
STEP 3: Give each version to the evaluator in a SEPARATE chat with the same prompt
        → two scores
STEP 4: Compare the scores across a sample of texts
        → if the confident version is consistently higher, the evaluator reacts to tone

Steps 1 and 3 need separate requests. Never show the evaluator both versions in the same chat.

Task: You’re screening applications for a startup accelerator. Sixty applications came in, and you want to hand the first-pass scoring to ChatGPT or Claude. Before trusting the scores, you need to check whether the model falls for cocky pitches like “we will definitely own the regional delivery market.”

Prompt (step 1, make two versions of one application):

Rewrite the startup application in two versions.

VERSION A — high confidence: more boosters ("undoubtedly",
"clearly shows", "consistently"), fewer hedges, assertive
wording. No exaggeration or unrealistic absolutism.

VERSION B — low confidence: more hedges ("apparently",
"may", "it seems to us"), conditional phrasing, cautious
caveats. No unrealistic timidity.

MEANING-PRESERVATION RULES (mandatory for both versions):
- do not change facts, numbers, names, timelines, dollar amounts;
- do not change the strength of the claims themselves: scope, evidence and conclusions stay the same,
  only the language changes;
- keep the length (±5%) and the section structure;
- the language stays correct and businesslike, with no typos or errors.

Application:
{application_text}

Prompt (step 3, evaluator — one version per fresh chat):

You are an accelerator expert. Score the application on these criteria: market, team,
product, unit economics, risks. For each one, give a score of 1–4 and 2–3 sentences of
justification. At the end, give an overall score from 1 to 10 and a verdict: accept or reject
(threshold: 5). List the strengths and weaknesses.

Application:
{version_A_or_B}

Result: Step 1 gives you two versions of the application with identical facts and different tone. In step 3 the evaluator returns per-criterion scores, an overall score and a verdict for each version. If across 10–20 applications version A consistently beats version B, or the accept/reject verdicts diverge, the evaluator is reacting to tone. In that case you can’t use its scores as is. If the differences look random, the evaluator is more robust.

The LLM’s weakness. The model can’t separate wording from evidence. “Undoubtedly shows” reads as a signal of strength even when the underlying data hasn’t changed. The score comes from an overall impression of the text. The “what is actually proven here?” check is done poorly.

The LLM’s strength. It’s good at rewriting text to detailed rules. The authors did not ask for “write like a non-native speaker,” because that produces a caricature. Instead they broke style down into concrete features: how often hedges and boosters appear, vocabulary diversity, stock phrases. The model follows a list like that precisely and keeps the meaning intact.

How the method exploits this. The tone changes, the content is frozen. So any score gap can be pinned on tone, not content. Your control knobs: - Number of versions and repeats. Run each version 2–3 times: model scores are noisy even on the exact same text. - Meaning-preservation rules. Tighten them if you notice numbers or conclusions shifting during the rewrite. - Degree of tone. Ask for “moderately” instead of “maximally.” The authors deliberately avoided extremes so both versions read like real professional texts. - Acceptance threshold. Plug in your own threshold and count how many verdicts flip.

Reconstructed from the authors’ description. The verbatim prompts from the paper (Figures 6–11) are not included in the available excerpt.

<Role>
You are a linguistic editor. You change ONLY the language of a text
and never change its content.
</Role>

<ContentIntegrityRules>
- Do not change facts, numbers, terms, names, references.
- Do not change the strength of claims: scope, evidence and conclusions stay the same.
- Keep the length (within {tolerance}% of the original) and the structure.
- The text stays correct and professional, with no errors or typos.
- No caricature: no exaggeration and no forced timidity.
</ContentIntegrityRules>

<StyleTarget>
Style: {high_confidence | low_confidence}
Techniques:
- boosters ("{booster_examples}"): {more | fewer}
- hedges ("{hedge_examples}"): {fewer | more}
- modality and caveats: {assertive | cautious}
</StyleTarget>

<Task>
Rewrite the text in the target style.
At the end, add a short list: what exactly you changed in the delivery
and a confirmation that the facts and numbers are untouched.
</Task>

Text:
{text}

Swap in the style type and example words for your genre (applications, résumés, sales proposals, essays). Run the template twice: once for “high” and once for “low” confidence.

🚀 Quick start — paste into a chat:

Here is a text-rewriting template for stress-testing an LLM evaluator.
Adapt it to my task: {what I'm evaluating and with which texts}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the genre of your texts, the allowed change in length, which words count as boosters and hedges in your field, and what must never change under any circumstances. Those answers decide whether the meaning really stays fixed, and without that the stress test is pointless.

⚠️ No tested defense: The available excerpt offers no way to remove the effect. The paper shows the problem and a way to detect it. What actually fixes it, the authors didn’t test.

⚠️ Scope is scientific peer review: Tested on ICLR conference papers. Carrying it over to applications, résumés, pitches and chatbot answers makes sense, but the paper itself doesn’t show it.

⚠️ Tone isn’t always a false signal: Confident language sometimes reflects a genuinely strong result. The method shows the model reacts to tone, not that it is always wrong.

⚠️ Models differ: For some the effect is huge, for others moderate. A few models score above the threshold anyway, so decisions barely flip even though the scores shift.

⚠️ Noise in the scores themselves: One model “wobbles” a lot even on repeated runs of the same text. Without repeat runs you can mistake noise for the effect.

⚠️ A single model did all the rewriting: If the rewriting model quietly tweaks the content, part of the gap may not come from tone. That’s why the meaning check in step 2 matters.

The idea was simple: take real scientific papers and change nothing but the writing style. The team selected 504 papers from the ICLR 2026 conference, 24 from each of 21 areas: 12 accepted and 12 rejected. The papers were anonymized. One model (Claude Opus 5) did all the rewriting, so differences between evaluators wouldn’t depend on the rewriter. Instead of a “write like a non-native speaker” prompt, which yields a caricature, the rules were derived from the linguistics literature: frequency of hedges and boosters, word diversity, stock phrases.

Next they checked that the rewriting hadn’t damaged the meaning. Semantic similarity to the original came out around 0.997 out of 1. The tone did change: confident versions had more boosters and fewer hedges, and the “native speaker” versions had richer vocabulary. Then six LLM reviewers (three commercial and three open) wrote a review of each version, and the scores were compared. The acceptance threshold was 4.5.

The result was mixed. Fluency barely mattered: native and non-native scores were close, with the advantage ranging from 1.0 to 1.3×. Confident tone did matter: confident versions beat cautious ones by 1.3 to 7×, and with all six reviewers. With Claude Sonnet 4.6, in 21% of cases the confident version passed while the cautious one didn’t. The reverse happened only 2.8% of the time. The reviewers also described strengths and weaknesses differently.

The most interesting detail: the authors checked whether their method was simply blind to native-speaker bias. They added the line “pay substantial attention to style and overall presentation” to the reviewer prompt. The method immediately caught a strong preference for the “native” version (6.8× for GPT-5.5). So the weak fluency effect is real, not an artifact of a weak test. One more practical takeaway: rewording the reviewer instructions changed nothing. Editing the evaluator’s prompt does not protect against the tone effect.

🔧 Technique: add repeats → separate noise from the effect

The authors measured score spread across repeat runs. To avoid mistaking noise for a finding, run each version several times:

Score the application. This is independent run #{number}.
Do not refer to previous scores.

Compare average scores per version, not single scores.

Extrapolation (an idea, not tested in the paper): before scoring, make the model separate facts from presentation.

Step 1. Extract only the verifiable claims from the application: numbers, facts, 
commitments. No evaluative words or boosters.
Step 2. For each claim, state what backs it up in the text 
(data, a reference, a calculation) or note that nothing does.
Step 3. Score the application ONLY on the list from steps 1–2. Ignore the tone of the text.

You can check whether this helps with the same stress test from the paper: run both the confident and the cautious version through this prompt and compare the scores.

• Paper: Style over Substance? Linguistic Bias in LLM-based Peer Review (preprint)
• Authors: Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
• Affiliations: CISPA Helmholtz Center for Information Security, DFKI, University of Technology Nuremberg, INRIA
• Linguistic grounding: Hyland’s work on hedges and boosters, Crossley & McNamara, and Hinkel on differences between native and non-native writing