Multilingual RAG: Documents and Questions in Different Languages
Multilingual RAG means a user can ask a question in one language and get a grounded answer from documents written in another, or in several others at once. It works when retrieval can match meaning across languages, not just words, and when the generation step is told clearly which language to answer in. Get either piece wrong and you get confident answers built on the wrong passages.
What "multilingual" actually means in RAG
There are two separate problems hiding under one label, and conflating them is the most common mistake teams make. The first is multilingual content: your knowledge base has documents in English, Spanish, French and German, and each document should be findable regardless of which language it's in. The second is cross-lingual querying: a user types a question in Spanish and the best matching passage happens to be in an English PDF. A system can solve the first without solving the second, and many "multilingual" products only do that: they index each language fine on its own but never bridge the gap between a query in one language and a document in another.
If your documents and your users always share the same language (all French documents, all French questions), you don't need cross-lingual retrieval at all, just a solid per-language pipeline. If you have a mix, for example a US parent company with policy documents in English but a workforce that also asks questions in Spanish, you need the cross-lingual piece specifically. Before building anything, write down which of these two situations you actually have, because the architecture differs.
Why cross-lingual retrieval is harder than it looks
Classic keyword search (BM25 and similar) matches tokens, so a French question about "résiliation de contrat" will never match an English passage about "contract termination" no matter how relevant that passage is: the words don't overlap. This is the core reason naive multilingual search fails. Hybrid search, which combines full-text matching with semantic vectors, only helps across languages if the vector model itself was trained to place equivalent meanings from different languages close together in embedding space.
That's the real technical requirement: a multilingual embedding model, not just a multilingual tokenizer. Models like multilingual E5, LaBSE, or OpenAI's and Cohere's multilingual embedding families are trained on parallel and comparable corpora specifically so that "How do I cancel my subscription?" and "¿Cómo cancelo mi suscripción?" land near each other in vector space even though no words match. If your pipeline uses an embedding model trained mostly on English text, cross-lingual recall will quietly degrade: queries in non-English languages will retrieve the wrong passages, or nothing at all, without any obvious error in the logs. For background on how embeddings represent meaning as numbers, see Embeddings Explained for Non-Specialists and Semantic Search Explained.
The silent failure mode
Cross-lingual retrieval failures rarely throw errors. The system returns an answer, often a fluent one, built from the nearest passages it could find, even if those passages are in the wrong language or only loosely related. Always check retrieval quality per language, not just overall.
Three architectures for multilingual RAG
There is no single correct approach. The right one depends on how many languages you support, how much the document set changes over time, and your tolerance for translation cost and latency.
Multilingual RAG approaches compared
| Approach | How it works | Best for | Main risk |
|---|---|---|---|
| Multilingual embeddings only | Index documents as-is; rely on a cross-lingual embedding model to match meaning across languages at query time | Mixed-language corpora, many languages, low maintenance budget | Lower precision than same-language matching; needs testing per language pair |
| Query translation | Translate the incoming question into the document's dominant language before retrieval, then answer in the user's original language | One main document language (e.g. English manuals) with questions in many languages | Translation errors shift meaning before retrieval even starts |
| Document translation at ingest | Translate every document into each supported language and index each version separately | Small, stable document sets, few target languages, high accuracy requirements | Expensive to maintain; translations go stale when source documents change |
In practice, most production systems combine the first two: multilingual embeddings for baseline cross-lingual recall, plus a lightweight query translation step used as a fallback or a re-ranking signal when confidence is low. Full document translation at ingest is usually reserved for a small number of critical documents (a safety manual, a core policy) where the cost of a mistranslation is high and the document doesn't change often.
Where translation happens matters
Translating at query time (the question) is cheaper and keeps the document corpus authoritative: you never have to maintain a translated copy that can drift out of sync with the source. Translating at ingest time (the documents) gives better retrieval precision because the match happens in the same language on both sides, but it multiplies your indexing cost by the number of languages and creates a maintenance burden every time a source document is updated. If you're estimating what a multilingual deployment will cost, read What Does RAG Really Cost? A Full Cost Breakdown before committing to full document translation across many languages.
Stemming, tokenization and hybrid search across languages
Even once cross-lingual embeddings are in place, the keyword half of hybrid search needs per-language handling. Stemming (reducing "running", "runs", "ran" to a common root) and stop-word removal are language-specific: an English stemmer applied to French text will mangle accented words and miss French-specific suffixes entirely, which quietly weakens the full-text side of hybrid search for every non-English document. The practical fix is to detect each document's language at ingest and apply a language-appropriate analyzer (most search engines, including Elasticsearch and PostgreSQL's full-text search, ship with stemmers for a dozen or more languages out of the box).
- Detect and store the language of each document (and each chunk, if documents mix languages) at ingest time
- Apply the matching stemmer and stop-word list per chunk rather than one global analyzer for the whole index
- Keep the cross-lingual embedding index separate from the per-language keyword index, and merge results at query time
- Normalize diacritics and case consistently, but don't strip them entirely: "résumé" and "resume" are different words in different contexts
- Re-check tokenization whenever you add a new language, not just at launch
Chunking strategy also interacts with language. Sentence and paragraph boundaries differ across languages (German compound words, Chinese and Japanese without spaces between words, right-to-left scripts like Arabic), so a chunker tuned for English punctuation can silently produce broken chunks in other languages. See Chunking Strategies for RAG for how fixed-size, semantic and structure-aware splitting behave differently, and test chunk boundaries on a real sample of each language you support rather than assuming the same settings work everywhere.
Quality checks: how to evaluate cross-lingual RAG
Standard RAG evaluation metrics (recall, faithfulness, citation accuracy, covered in How to Evaluate a RAG System) still apply, but they need to be measured per language pair, not as a single blended number. A system that performs well overall can still be quietly broken for one language because that language is underrepresented in your test set.
- Build a test set with questions in every language you support, each with a known correct source passage, even if that passage is in a different language than the question
- Measure retrieval recall separately for same-language queries and cross-lingual queries; a meaningful gap between the two tells you the embedding model is the weak point
- Check that the generated answer is actually in the language the user asked in, not the language of the retrieved passage (this is a common and easy-to-fix bug: the model defaults to the source document's language)
- Verify citations point to the real passage and not to a mistranslated paraphrase of it
- Re-test after any change to the embedding model, the chunking settings, or the addition of a new document language
One subtle failure worth testing explicitly: answer language drift. Some language models will answer in the language of the retrieved context rather than the language of the question, especially when the context dominates the prompt. If you need the answer language locked to the question language regardless of source material, say so explicitly in the system prompt and verify it in your test set rather than assuming it.
A practical rollout checklist
If you're adding a second or third language to an existing knowledge base, a staged rollout reduces risk compared to flipping on every language at once.
- Start with the language pair that matters most to your users, measure it in isolation, then expand
- Confirm your embedding model's documentation explicitly lists cross-lingual training, not just multilingual tokenization support
- Keep a small set of "known answer" questions per language to catch regressions before they reach users
- Decide upfront whether answers must stay in the question's language or may mirror the source document, and encode that decision into the prompt
- Document which languages are officially supported; an unsupported language that happens to work today can silently break with the next model update
Kopik's hybrid search combines full-text and semantic matching automatically when you upload documents through the catalogue or create a new base, and the same base can hold documents in multiple languages. If you're building a product on top of this rather than using the website directly, the developer docs cover both the REST API and the MCP server, which lets AI clients like Claude, Cursor or ChatGPT query a multilingual base the same way a human would through the chat interface.
Test cross-lingual retrieval before you commit to an architecture
Upload a small multilingual sample, ask questions in a different language than the source documents, and see how retrieval actually performs before building a custom pipeline.
If your use case is closer to long documents in a single pair of languages, it's also worth comparing retrieval against simply feeding more context to a long-context model; RAG vs Long-Context Models covers when each approach makes sense, and the trade-offs don't disappear just because multiple languages are involved; if anything, retrieval's cost advantage grows as the corpus grows across languages.
Frequently asked questions
Do I need a separate embedding model for each language?
No, and in most cases you shouldn't. A single multilingual embedding model trained for cross-lingual retrieval (such as multilingual E5 or LaBSE) places equivalent meanings from different languages close together in the same vector space, which is exactly what you need for cross-lingual search. Separate per-language models complicate merging results and don't solve the cross-lingual matching problem on their own.
Should I translate my documents or translate the user's question?
Translating the question at query time is cheaper and avoids maintaining translated copies of documents that can drift out of sync with the source. Translating documents at ingest gives slightly better precision but multiplies indexing cost and maintenance effort by the number of languages. Most teams use multilingual embeddings as the baseline and reserve full document translation for a small number of high-stakes documents.
Why does my RAG system answer in the wrong language?
This usually happens when the generation prompt doesn't explicitly instruct the model to answer in the question's language, so it defaults to the language of the retrieved passages instead. Fix it by adding an explicit instruction in the system prompt and testing it as part of your evaluation set, not assuming it works by default.
Does hybrid search work across languages automatically?
Only the semantic half does, and only if the embedding model was trained for cross-lingual retrieval. The full-text (keyword) half requires a language-appropriate stemmer and stop-word list for each language in your corpus; a single English analyzer applied to all documents will weaken keyword matching for every other language.
How do I test whether my multilingual RAG system actually works?
Build a small evaluation set with questions in every supported language, each paired with a known correct passage, and measure retrieval recall separately for same-language and cross-lingual queries. A large gap between the two numbers tells you the embedding model, not the generation step, is the weak point.
Can one knowledge base mix documents in several languages?
Yes. A single base can hold PDFs, Word files or Markdown in multiple languages as long as the indexing pipeline detects each chunk's language and applies the right stemming, and the embedding model supports cross-lingual matching. This is how Kopik indexes mixed-language uploads without requiring separate bases per language.
Get the Kopik newsletter
New knowledge bases, RAG guides and product news. One email every week or two, unsubscribe in one click.
By subscribing you agree to receive our newsletter. We never share your address.