AI system design interview: design a multilingual assistant
Detect the language of each message, use a multilingual embedding model so questions in one language can find documents in another (or translate the query for retrieval), and answer in the user's language. Budget for higher token counts in languages the tokenizer handles poorly, evaluate each language separately with native speakers, and route languages where your main model is weak to one that does better.
Question
Your assistant must serve customers in English, Spanish and Hindi, with documents mostly in English.
Clarify
- Traffic share per language.
- Whether documents exist in each language or only in English.
- Mixed-language messages, like Hindi written in Latin script.
- Quality bar per language.
Architecture
| Stage | Component |
|---|---|
| Detection | Language per message, including transliterated text |
| Retrieval | Multilingual embeddings over English documents, or query translated to English |
| Generation | Answer in the user's language, citations to the English source |
| Routing | Per-language model choice where quality differs |
| Evaluation | Per-language test sets labelled by native speakers |
Deep dive: token cost
Tokenizers are trained mostly on English, so the same sentence can take several times more tokens in another language. That raises cost, fills the context window faster and can increase latency, so measure tokens per message per language with your model's tokenizer and use the numbers in your cost model.
Failure modes
| Failure | Mitigation |
|---|---|
| Answer in the wrong language | Language detection, explicit instruction, output check |
| Poor retrieval across languages | Multilingual embeddings tested per language, or query translation |
| Mistranslated policy terms | Glossary of key terms per language |
| Quality gap hidden by English-heavy averages | Report metrics per language |
Evals
A test set per language with native speakers labelling a sample, reported separately, because quality often differs a lot between languages.
What interviewers listen for
Per-language evaluation, token cost in the estimate, and a clear choice for cross-language retrieval.
From the floor
In the inference pipeline lecture, I explained why other languages cost more: a word that's one token in English might take four tokens in Hindi, because byte pair encoding and similar tokenizers were designed mostly for English.