Semantic Search Explained: Meaning-Based Search vs Keywords

Semantic search finds documents and passages based on meaning rather than exact word matches, using numerical representations of text called embeddings. A query like "staff holiday entitlement" can retrieve a passage titled "annual leave policy" even though the two phrases share almost no words. It underpins most modern RAG and AI assistant tools, but it is not a drop-in replacement for keyword search: each approach has failure modes the other does not. This guide explains how semantic search actually works, what query expansion and multilingual search add, and when you still need keyword matching alongside it.

Keyword search vs semantic search: what's actually different

Traditional keyword search (sometimes called full-text or lexical search, built on algorithms such as BM25) matches the literal tokens in your query against the tokens in your documents, with some stemming and stop-word removal. It is fast, predictable and excellent at exact matches: product codes, names, error messages, legal citations. Its weakness is synonymy: if the document says "termination of employment" and the user searches "getting fired", a pure keyword engine often finds nothing, because there is no shared vocabulary between the two phrases.

Semantic search works differently. It converts both the query and the document chunks into embeddings, dense vectors that place similar meanings close together in a high-dimensional space, and then ranks results by vector similarity (usually cosine similarity) rather than word overlap. For a plain-English introduction to how that conversion happens, see Text Embeddings Explained. The practical effect is that semantic search tolerates paraphrasing, informal language and conceptual queries far better than keyword search does.

How semantic search actually works under the hood

A semantic search pipeline has three moving parts. First, your source documents are split into chunks, paragraphs or sections small enough to embed meaningfully but large enough to retain context. Chunk boundaries matter more than most people expect: splitting mid-sentence or mixing two unrelated topics in one chunk degrades retrieval quality even if the embedding model itself is excellent. We cover the trade-offs in Chunking Strategies for RAG. Second, each chunk is passed through an embedding model and stored as a vector, typically in a vector index built for fast approximate nearest-neighbour lookup. Third, at query time the user's question is embedded the same way, and the system returns the chunks whose vectors are closest to the query vector.

Why similarity scores aren't the whole story

A high similarity score tells you two pieces of text are topically related, not that one correctly answers the other. A question about "cancelling a direct debit" and a passage about "setting up a direct debit" can sit close together in vector space because they share so much vocabulary and context, yet one is the opposite of what the user wants. This is why production systems layer a re-ranking step, or combine semantic scores with keyword signals, rather than trusting raw vector similarity alone.

Query expansion: making the search smarter

Query expansion means broadening or rewriting a user's query before searching, so the system considers terms the user did not type but clearly meant. There are two common flavours. Lexical expansion adds synonyms and related terms ("holiday" expands to include "annual leave", "PTO", "time off") and can be layered on top of a keyword engine. Semantic expansion goes further: the system can generate a short hypothetical answer to the query and embed that instead of (or alongside) the raw question, a technique sometimes called hypothetical document embeddings, because an imagined answer often resembles the real passage more closely than the bare question does.

Query expansion matters most when users phrase things casually or incompletely. A query such as "can I bring my dog to the office" needs to match a policy document that never uses the word "dog" at all, only "pets" or "animals on site". Good expansion closes that gap without forcing every document author to anticipate every possible phrasing.

Multilingual semantic search

Because embeddings represent meaning rather than specific words, multilingual embedding models can place a French query and an English passage close together in the same vector space if they mean roughly the same thing. This makes cross-language retrieval genuinely practical: a UK team can upload English-language policies and still get sensible answers to questions asked in Welsh, Polish or Urdu, without maintaining separate translated document sets. In practice, quality varies by language pair and by domain. Common-language pairs with lots of training data (English/French, English/Spanish) tend to work well; low-resource languages and highly technical jargon (clinical terminology, legal Latin, engineering part names) are weaker, because the embedding model has seen fewer examples of that vocabulary used in context.

When semantic search fails

Semantic search is not a universal upgrade. It systematically struggles with certain query types, and knowing them helps you decide when to fall back to keyword matching or add a safety net.

• Exact identifiers: order numbers, part codes, case references, invoice numbers. These need literal matching, not meaning.
• Negation: "policy that does not require a deposit" can retrieve the opposite passage, because "deposit" dominates the embedding regardless of "not".
• Numeric thresholds: "contracts under £10,000" is a filter, not a meaning, and most embedding models do not reliably encode numeric comparisons.
• Rare or newly coined terms: product names launched last month, internal codenames, or niche acronyms the embedding model never saw during training.
• Very short, ambiguous queries: a single word like "return" could mean a product return, a tax return or a function return, and the embedding alone cannot disambiguate without more context.
• Near-miss topical matches: passages that are semantically adjacent but factually wrong, such as retrieving the "2023 pricing" page for a query about "2026 pricing".

Hybrid search: the practical answer

Most production systems that need to be reliable use hybrid search: running keyword (BM25-style) search and semantic search in parallel, then merging and re-ranking the results. This captures the best of both: exact matches surface when they exist, and conceptual matches surface when the wording differs. It is a more robust default than either approach alone, and it is the architecture we recommend in Hybrid Search: Why BM25 Plus Vectors Beats Either Alone for RAG. If you are deciding between retrieval and simply pasting documents into a long-context model instead, RAG vs Long-Context Models walks through that trade-off separately.

Building semantic search for your own documents

Setting up semantic search from scratch means choosing an embedding model, a vector index, a chunking strategy, and ideally a keyword layer and re-ranker on top, then maintaining all of it as documents change. For most teams and consultants, that is considerably more infrastructure than the task warrants. On Kopik, you upload your documents (PDF, Word, text or Markdown), and the platform handles chunking, embedding and indexing for hybrid search automatically, combining full-text matching with semantic keyword expansion rather than relying on vector similarity alone. You can keep a base private for internal use, or list it publicly in the catalogue and charge per question, keeping 70% of what it earns. Answers come back grounded in your documents with cited passages, or as raw passages only if you prefer to build your own answer layer. You can query it from the website, through a REST API with API keys, or via an MCP server from tools such as Claude, Cursor or ChatGPT, details are in the developer docs.

A quick checklist before you rely on semantic search

1. Test with real user phrasing, not polished internal terminology, since that is how semantic search earns its keep.
2. Check exact-match cases separately: order numbers, policy clause references, part codes.
3. Confirm chunk sizes don't split key facts across two chunks; see Chunking Strategies for RAG for guidance.
4. Add a keyword fallback or hybrid ranking rather than trusting vector similarity alone.
5. If you operate in more than one language, test multilingual queries with your own jargon, not generic examples.
6. Review a sample of cited passages for faithfulness; see How to Evaluate a RAG System for metrics worth tracking.

If you are building an AI agent or assistant on top of search results rather than serving them directly to a human, the reliability bar is higher again: an agent that silently acts on a near-miss retrieval can cause real harm. That topic is covered in Why AI Agents Need Reliable Knowledge Sources, and the broader terminology used throughout this field is collected in the AI and LLM Glossary for Professionals.