Multilingual RAG: Querying Documents in a Different Language From Your Questions

Multilingual RAG means building a retrieval-augmented system where the question, the source documents, or both, can be in different languages, and still returning the right passage with a faithful answer. It works by combining cross-lingual embeddings (which place similar meanings close together regardless of language) with language-aware keyword search, and it needs explicit testing because failures are often silent: the system returns an answer, just not from the right document.

Why mixing languages breaks ordinary RAG

A standard RAG pipeline chunks documents, indexes them with embeddings and a keyword index, then matches a question against that index. If your knowledge base is monolingual, this works well out of the box. The trouble starts the moment your organisation has, say, English policy documents, a French subsidiary's contracts, and a German supplier's technical specs, and someone wants to ask one question across all of them.

Two separate problems stack up. First, keyword search (BM25 and similar) matches literal tokens, so a question in English will simply never match a French document on lexical overlap alone, aside from shared proper nouns or numbers. Second, even embedding-based semantic search is only as cross-lingual as the embedding model you chose: many popular models are trained mostly on English text and place other languages in a different, poorly aligned region of the vector space. The result is a system that looks fine in English-only demos and then quietly fails the moment a real multilingual question arrives.

If you haven't yet, it helps to understand the retrieval mechanics first. We cover the basics in Semantic Search Explained: Meaning-Based Search vs Keywords and the underlying vector mathematics in Text Embeddings Explained: A Plain-English Guide for Non-Specialists.

Three architectures for cross-lingual retrieval

There isn't one correct way to handle multilingual retrieval. Most production systems use one of three approaches, often blended.

1. Translate the query at retrieval time

The question is machine-translated into the dominant language (or languages) of the corpus before it hits the search index, then the retrieved passages are shown in their original language with the final answer generated in the user's language. This is cheap, fast, and doesn't require touching the documents. It works well when your corpus is mostly one non-English language and your users mostly ask in English, or vice versa.

The weakness: translation quality varies by domain. A generic translator can mangle legal, medical or engineering terminology, which then sends the search in the wrong direction. If your documents are full of jargon (HMRC terminology, ISO standard clauses, clinical codes), test the translation step specifically against those terms, not against everyday sentences.

2. Translate or duplicate the documents at ingestion time

Instead of translating questions on the fly, you translate the documents once, during ingestion, and index both the original and translated versions. This front-loads the translation cost but means retrieval never depends on the quality of a live, per-question translation, and you can run quality checks on the translated corpus once rather than on every query. It suits knowledge bases that get asked the same handful of languages repeatedly, where the one-off cost is worth it. The tradeoff is storage and maintenance: every document update now needs re-translation, and chunking decisions (see Chunking Strategies for RAG) have to be applied consistently to both versions so citations still point to a coherent passage.

3. Multilingual embeddings with no translation at all

Modern multilingual embedding models are trained so that semantically equivalent sentences in different languages land near each other in vector space, without any translation step. A question in English can retrieve a passage in Spanish directly on semantic similarity. This is elegant and avoids translation cost and latency entirely, but it has a real limitation worth testing before you rely on it: alignment quality drops for low-resource languages, for very short or jargon-heavy text, and for exact figures, dates, clause numbers and names, which embeddings don't represent precisely. A multilingual embedding model is very good at finding

"the paragraph about termination notice periods" across languages, and much less reliable at finding the one clause that says "30 days" rather than "60 days", because the surrounding semantics look identical.

Stemming, tokenisation and hybrid search per language

Full-text search isn't language-neutral. Stemming (reducing "regulations", "regulated" and "regulating" to a common root) and stop-word removal depend entirely on the language's grammar. A French-language index needs French stemming rules, a German index needs to handle compound words (a single German word can bundle several English concepts together), and languages without clear word boundaries, like Japanese or Chinese, need proper segmentation before any keyword index makes sense at all.

This matters because hybrid search, the combination of keyword (lexical) search and vector (semantic) search, consistently outperforms either technique alone for RAG, as we explain in Hybrid Search: Why BM25 Plus Vectors Beats Either Alone for RAG. But that advantage only holds if the lexical half is actually configured for the language in question. Running an English stemmer against French text, or none at all against German compounds, gives you a weak keyword layer that drags down the overall hybrid score rather than lifting it.

• Detect the language of each document (and ideally each chunk, for mixed-language files) at ingestion time rather than assuming one language for the whole base.
• Apply language-specific stemming and stop-word lists per chunk, not a single global configuration.
• Keep numbers, dates, currency symbols, clause references and proper nouns out of aggressive stemming, since these are exactly what a precise query is usually looking for.
• For compound-heavy languages, consider decompounding (splitting compounds into their parts) so that a search for one component still matches.

Quality checks that catch silent failures

The hardest part of multilingual RAG isn't building it, it's knowing when it's quietly wrong. A monolingual RAG system that retrieves the wrong passage usually produces an answer that's at least topically related, which is easy to spot in testing. A multilingual system can retrieve a passage in the wrong language entirely, or the right language but the wrong document, and still generate a fluent, confident-sounding answer because the generation step doesn't know the retrieval step failed.

These checks overlap heavily with the general discipline of RAG evaluation, which we cover in more depth in How to Evaluate a RAG System: Retrieval Metrics, Faithfulness and Citation Accuracy. The multilingual twist is that you need separate test sets per language pair, not one global test set, because performance genuinely varies by language and the failures don't generalise.

A practical checklist for building a multilingual knowledge base

1. Inventory your languages: list every language present in your source documents and every language your users will ask questions in, including ones you don't expect but should plan for (customer support tickets often arrive in more languages than marketing content).
2. Choose an embedding strategy that explicitly supports your language pairs, and verify it rather than assuming from the model's marketing description.
3. Decide where translation happens: query time, ingestion time, both, or not at all if your embeddings handle it well enough for your use case.
4. Configure language-specific stemming and tokenisation for the keyword half of hybrid search, per document or per chunk.
5. Build a small multilingual test set (even 20 to 30 question/answer pairs per language pair is a useful start) and measure retrieval accuracy before launch.
6. Keep citations visible so users can verify an answer even if they don't read the source language fluently, since machine translation of a cited passage is a useful fallback but shouldn't replace showing the original.

If your knowledge base already sits on hybrid search infrastructure built for a single language, the migration work is mostly in steps 2 to 4: adding language detection, swapping in multilingual-aware embeddings, and reconfiguring the lexical index per language rather than rebuilding everything from scratch.

Where this fits with RAG as a service

Building and maintaining this stack from scratch (language detection, per-language stemming, a multilingual embedding model, translation fallbacks, and the evaluation harness to check it all works) is a meaningful engineering commitment on top of the usual RAG pipeline. On Kopik, uploading a document for hybrid search handles indexing and chunking automatically, and because retrieval combines full-text and semantic matching by default, adding documents in a second or third language to an existing base is largely a matter of uploading them and checking the quality checks above hold up, rather than rebuilding your retrieval layer. For teams querying a base programmatically across languages, the developer documentation covers the REST API and MCP server options, both of which pass the question through as-is, leaving your application free to translate it first if that suits your architecture better.

If you're weighing up retrieval against simply pasting everything into a long context window regardless of language, the tradeoffs are similar to the monolingual case, just with translation and alignment costs added on both sides. See RAG vs Long-Context Models: When to Retrieve and When to Stuff the Prompt for the general argument.

Common pitfalls worth flagging to stakeholders

• Assuming one embedding model "does multilingual" without testing the specific language pairs you need, especially less common ones.
• Treating translation as a one-off task rather than something that needs re-running whenever source documents change.
• Skipping language-specific stemming because the keyword layer feels like a secondary detail next to the embedding model.
• Not testing with questions that hinge on exact figures, dates or clause numbers, where semantic similarity across languages is weakest.
• Hiding the source language from users entirely, which makes it harder for a bilingual reviewer to catch a retrieval error during an audit.

Multilingual RAG rewards the same discipline as any other RAG deployment: test retrieval quality explicitly, keep citations visible, and treat translation as an engineering component with its own failure modes rather than a solved problem. Get those basics right per language pair and the rest of the pipeline, chunking, generation, and answer formatting, works much as it would for a single-language knowledge base.