TL;DR
Models reply in the wrong language when a prompt mixes several languages — even when the context clearly signals which language is expected. Say a user writes a prompt in English and drops in a few Spanish words: the model answers entirely in Spanish, even though English was expected. Or the other way around: the instruction is in Spanish, the text to process is in English, and the model replies in English instead of Spanish.
The core problem is a strong bias toward the non-English language. As soon as a little text in another language shows up in the prompt (Korean, Chinese, Indonesian, and so on), models become far more likely to answer in that language, even when the prompt's main language and the context point to English. The gap is huge: from 96% success when Korean is expected down to 21-75% when English is expected. The cause: models lean on surface signals — the share of characters from each language in the prompt and the language of the last word (if the prompt ends with a non-English word, the reply will most likely be in that language). Chain-of-Thought doesn't help — it actually makes things worse.
A fix exists, but it's for model developers: Code-Switching Aware DPO with ~1000 examples dramatically improves things. For chat users — state the reply language explicitly whenever you mix languages in a prompt. Add at the end: "Reply in English" or "Responde en español". That sidesteps the problem almost completely.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
The Problem at a Glance
CASE 1: Dropping in words from another language (intra-sentential)
Prompt: "What is the reason for doing a calentamiento before the main workout?"
[English grammar + a Spanish noun]
Expected: English (the main grammatical language)
Actually: Often Spanish
CASE 2: Instruction and content in different languages (inter-sentential)
Prompt: "Aquí tienes un borrador de artículo en inglés: [text]. Encuentra los errores."
[instruction in Spanish + content in English]
Expected: Spanish (the instruction language, for comprehension tasks)
Actually: Often English
The Problem in Action
Task: You're analyzing a startup's launch strategy. You write the prompt in Spanish but use English terms (product-market fit, retention, churn) because that's what you're used to.
Prompt WITHOUT an explicit language:
Analiza esta go-to-market strategy para un B2B SaaS:
- Enfoque en clientes enterprise
- Customer acquisition cost de unos $5000
- Lifetime value estimado en $50000
¿Qué hay que considerar en los risk factors?
Result:
The model will very likely answer in English, because:
1. The last word of the prompt is English ("factors")
2. Latin-script English words make up a big chunk of the prompt (~40-50%)
3. Models are biased away from the primary language when languages are switched
Prompt WITH an explicit language:
Analiza esta go-to-market strategy para un B2B SaaS:
- Enfoque en clientes enterprise
- Customer acquisition cost de unos $5000
- Lifetime value estimado en $50000
¿Qué hay que considerar en los risk factors?
Responde en español.
Result: The model will answer in Spanish with ~97-100% accuracy (instead of 21-75% without the explicit instruction).
Why This Happens
Models don't analyze the deep structure of language — they rely on surface signals:
The share of characters from each language — the more Cyrillic, Hangul or Chinese characters in the prompt, the higher the chance the reply will be in that language. The model counts characters; it doesn't analyze grammatical structure.
The language of the last word — the strongest factor of all. If the prompt ends with a Korean, Chinese or Spanish word, the model becomes far more likely to answer in that language. This is recency bias — the most recent context outweighs everything else.
Bias toward the non-English language — if the prompt contains even a little text in another language, models pick it for the reply "by default". The likely reason: in the training data, code-switching examples more often had non-English answers.
Chain-of-Thought makes it worse — when you ask the model to "think out loud", accuracy DROPS (by up to -50% in some cases). That means the model has no internal criteria for figuring out the expected language — it can't "reason its way" to the right answer.
Post-training (SFT+RLHF/DPO) amplifies the bias — aligning models to user instructions paradoxically makes things worse for English in code-switched prompts. Most likely, the alignment data has few examples where mixed languages still call for an English answer.
Prompt Template
To guard against surprise language switches:
{your prompt with mixed languages}
Reply in {English/Spanish/another language}.
Examples:
- Reply in English.
- Responde en español.
- Reply in English only.
When you should always add it: - You use English terms in a prompt written in another language - You give the instruction in one language and the content to process in another - The last word of the prompt isn't in the language you want the answer in - "Foreign" characters make up >20-30% of the prompt
Why It Works
An explicit instruction bypasses the surface heuristics. When the reply language is stated outright, models follow the instruction with 96-100% accuracy (even the very same models that scored 21-75% without it). This proves the problem is NOT a lack of language ability — models CAN generate in any language. The problem is inferring the expected language from implicit signals.
Levers you control: - Explicitness — the more specific ("in Spanish"), the better. "Preferably in Spanish" works worse. - Position — at the end of the prompt is more effective (the last word carries a lot of weight). - Repetition — if the prompt is long and multi-step, repeat the language instruction in the middle and at the end.
Limitations
⚠️ Doesn't help against "raw" glitches: Stating the language does NOT protect against stray characters from a third language (say, sudden Japanese or Chinese characters in the middle of Spanish text). This problem shows up more often in models after DPO/RLHF training.
⚠️ Text quality suffers: Even when the model replies in the right language after an explicit instruction, the text can sound unnatural — literal translations instead of established loanwords, calques instead of idioms. Especially noticeable with an instruction in one language and content in English.
⚠️ The problem is systemic; the fix is a band-aid: The study shows the problem lies in how models are trained (not enough alignment data for code-switched contexts). An explicit instruction is a workaround, not a fix for the root cause.
Resources
OLA: Output Language Alignment in Code-Switched LLM Interactions
Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama, Alice Oh
KAIST
The OLA benchmark is available: github.com/juhyunohh/OLA