In This Handbook

Vector Databases and Semantic Retrieval

Traditional keyword search matches exact words, which often breaks down when dealing with natural language meaning. If a user’s query doesn’t use the exact phrasing present in a document, a pure keyword lookup may miss the relevant information entirely. Common failure modes include:

Synonym blindness: Keyword searches, such as for “doctoral candidates” or “battery safety technology,” are limited because they require exact word matches and may miss relevant documents using different terms (e.g., “PhD students”). Users must “think like the database,” leading to missed information when terminology or jargon varies.
Phrasing mismatch: Keyword search overfits specific wording, potentially missing relevant content that uses synonyms or alternate expressions. For instance, searching for “terminate agreement” may not find documents using “contract dissolution,” despite the identical meaning.
Language and formatting variance: An English-only keyword search won’t retrieve a relevant document written in Spanish. Similarly, differences in spelling or formatting (singular vs plural, hyphenation, etc.) can throw off strict term matching. Important information can hide in forms that a simple lexical search fails to recognize as related.
High noise from broad matches: To avoid missing synonyms, users may resort to long Boolean queries with many OR’ed terms. This often pulls in too many results, drowning the user in noise. For instance, searching a database for “Section 3 of the Companies Act” might return hundreds of documents that contain the words “Section” or “Companies Act” but aren’t the specific section needed. The precise intent (a particular section of a law) is lost amid all the loosely related hits.

In summary, keyword search is inadequate because it relies on literal text matching and fails to capture intent. For LLM-based systems needing relevant context, this brittle approach risks retrieving irrelevant snippets, leading to garbled answers or hallucinations. We require a retrieval mechanism that goes beyond exact matching to capture the query’s semantics.

ⓘ Keyword search matches words, not meaning

Traditional search engines operate on lexical overlap. They answer the question: “Do these strings match?” Semantic retrieval answers a different question: “Do these ideas match?” This distinction is why embeddings are necessary for LLM systems.

Embeddings as semantic representations

Modern semantic retrieval uses embeddings, which are numeric vectors, to represent text based on meaning instead of exact words. Similar texts have vectors close together in a high-dimensional “idea space.” For example, phrases like “laptop stand” and “notebook riser,” despite different words, are mapped closely because they share the same semantic intent. The embedding captures the concept and context, not just surface tokens.

Example: Same meaning, different words

Query: “ergonomic laptop stand”

Retrieved text: “Adjustable notebook riser improves posture”

Even though no exact words overlap, the embedding vectors are close in semantic space, allowing retrieval. A keyword search would likely fail here.

How are embeddings generated?

Embedding models (often transformer encoders) convert text into fixed-length vectors (e.g., 384 or 768 dimensions). The model is trained so texts with similar meaning have vectors pointing in similar directions, which is measurable using metrics like cosine similarity (1.0 being overlapping meaning). This vector space overcomes keyword matching issues; synonyms and translations are located near each other. Essentially, embeddings act as a semantic common ground, allowing retrieval systems to understand the user’s intended meaning, not just literal words.

ⓘ Embeddings power retrieval, not generation

Embedding models are usually separate from the LLM. Their sole job is to map text into a semantic space for retrieval. They do not generate answers, reason, or verify truth.

How semantic retrieval works

Semantic retrieval with embeddings operates through a pipeline that pairs with the LLM. The process typically works as follows:

Document encoding: The embedding model pre-processes all documents in your database, converting each document (or chunk) into an embedding vector, which is then stored in a vector index. This offline or periodic process indexes your knowledge. The resulting vectors, representing your content, are stored alongside a reference to the original text (and sometimes metadata like document ID or source).
Query encoding: When a user query comes in, we run the same embedding model on the query to get its vector representation. This query vector lives in the same vector space as the document vectors.
Similarity search: The system compares the query’s embedding to document embeddings using efficient nearest-neighbor search algorithms to find documents with similar meaning. Vector databases employ advanced indexing (e.g., HNSW graphs, product quantization) for rapid retrieval of the top K nearest neighbors, even among billions of vectors.
Retrieval of results: The system returns the top-ranked text chunks, those with the closest vectors to the query vector, as relevant context. This context provides the semantically relevant information the LLM uses to answer the question. Crucially, this retrieval is based on meaning rather than exact keywords. For instance, a query on “renewable energy sources” could retrieve text about “solar panels and wind turbines” because the embedding model understands the conceptual link, even if the word “renewable” is absent.
Augmentation of LLM context: The retrieved text chunks are then inserted into the LLM’s prompt (e.g. as a prepended context or as cited excerpts), so that the model has additional knowledge to generate a factual, contextually-grounded answer. This combination of vector-based retrieval with the generative model is what we call Retrieval-Augmented Generation (RAG).

Behind the scenes, a vector database or similarity search library is responsible for steps 1–3. It handles storing the embeddings and performing fast similarity lookups. This is quite different from a traditional relational database or a search engine inverted index, and it’s why specialized vector DBs have become popular in AI applications.

Example: A semantic retrieval trace

User query: “What are safe battery storage practices?”

Query is embedded into a vector
Vector DB retrieves chunks about thermal runaway, lithium-ion storage, and fire safety
These chunks are injected into the prompt
The LLM generates an answer grounded in those excerpts

The LLM never “knows” battery safety on its own. It only uses what retrieval supplies.

What vector databases actually store

A vector database is a database optimized for storing and querying vectors in high-dimensional space. At its core, the vector DB holds a collection of embedding vectors (often hundreds of dimensions each), along with identifiers that tie each vector back to the original content (text, image, etc.) from which it was generated. It’s designed to answer questions like “Which stored vectors are most similar to my query vector?” very efficiently.

So what’s inside the vector DB? Primarily:

The vectors themselves: A vector database stores the numerical coordinates (embeddings) for each piece of data, often using an index structure and Approximate Nearest Neighbor (ANN) algorithms (like HNSW, IVF, or PQ) for fast similarity searches, even with large volumes. Conceptually, it’s a list of vector space points, though compression or quantization may be used instead of storing every raw vector..
Metadata and IDs: Alongside each vector, the DB will typically store an ID or reference, plus any metadata. For example, metadata could include the source document title, a timestamp, tags, or other filters. This allows filtering search results (e.g., only retrieve vectors from documents of a certain type or date) and helps with interpreting results. The ID lets the system fetch the actual text of the document when a match is found.
Index structures: Vector DBs use indexes (like graph-based, tree-based, or quantization-based) for rapid similarity computations, avoiding a brute-force search. This involves storing precomputed data structures that increase query speed at the cost of some storage overhead.

ⓘ Vector databases do not store knowledge

Vector DBs store numerical representations of text, not facts, logic, or truth. They enable similarity search, not understanding.

Vector databases (VDBs) store numerical embeddings and associated references, enabling semantic retrieval through similarity search. Unlike knowledge graphs, VDBs don’t inherently store structured schemas or concept relationships. Popular VDBs like Pinecone, Weaviate, Milvus, and Chroma are optimized for fast, scalable lookups and handle dynamic updates. VDBs are crucial for real-time semantic search and feeding relevant context to LLMs from large datasets.

Trade-offs in vector retrieval

Semantic retrieval via vectors is powerful, but it comes with trade-offs and considerations. It’s not a silver bullet for every scenario, and in some cases combining it with other methods yields the best results. Key trade-offs include:

Semantic recall vs. precision: While embedding search (vector search) offers high recall by retrieving documents based on meaning, even if loosely relevant, it can lack precision, often returning general context instead of the specific fact needed (e.g., getting the Act but missing “Section 3”). Keyword search is precise but often retrieves irrelevant “junk.” Hybrid search combines the high recall of vector search with the pinpoint precision of keyword search to address this limitation.
Handling exact matches and rare terms: SVector models may struggle with very specific proper nouns or codes (like product IDs or error messages), sometimes treating them as unknown tokens and finding semantically similar context instead of the exact string. For example, an embedding might map an unseen error code “X200” to “X201.” Hybrid retrieval combines lexical (keyword) and semantic (vector) search to prevent overlooking exact string matches, typically by merging or reranking the results of both search types.
Performance and scalability: Vector databases face challenges: storing high-dimensional vectors is memory-intensive, and searching can be slower than inverted index lookups, especially for high accuracy (recall). Many use approximate search to boost speed by slightly sacrificing accuracy, but this risks missing the true nearest neighbor. Users must tune index parameters for an acceptable recall level. If maximum completeness is essential (e.g., legal discovery), smaller vectors or brute-force search on a limited set may be required, incurring performance costs. Vector retrieval thus necessitates balancing speed and result completeness.

Embedding quality and domain shift: Vector search quality depends on the embedding model. For domain-specific content, general models may fail, requiring specialized embeddings or fine-tuning to prevent irrelevant results (misfires). A key challenge is embedding drift: updating the model changes document representations, necessitating costly re-indexing of the entire corpus. This drawback not shared by keyword indexes. This coupling of index to model means semantic improvements can incur high re-embedding costs.

Difficulty with structured queries: Vector search excels at “fuzzy” natural language queries (for semantic similarity), finding conceptually related text. However, for highly specific or structured information (e.g., “which database has exactly 5TB of storage free?”), a vector approach may struggle with exact numbers or filters. Structured queries or databases handle these precisely, unlike vectors, which are not suited for calculations, comparisons, or exact filtering. Graph-based retrieval, discussed later, shows how structured representations can complement vectors.

To mitigate these trade-offs, advanced RAG systems often employ multi-step retrieval techniques. For instance, a system might do an initial vector search to get relevant documents, then run a keyword search within those documents to locate a precise answer. Or use a reranker model to sort the vector results by how likely they contain the answer (improving precision). These hybrid approaches acknowledge that while vector semantic search is powerful, it isn’t foolproof. The theme is: use semantic retrieval as the first pass to broadly understand relevance, then refine. We’ll delve more into those strategies (chunking, re-ranking, multi-hop queries) in a later lesson.

Clarifying critical misconceptions

Semantic vector search has become a hot topic, and with it come a few misconceptions that are important to clarify:

Vectors encode truth: Vectors encode similarity, not truth. Nearest neighbors are texts written similarly or about similar concepts, which doesn’t guarantee factual correctness. A vector search might retrieve an outdated or mistaken claim (e.g., an old CEO’s name), leading the LLM to output a confident but wrong answer. Similarity is a pattern-match, not a truth endorsement.
Semantic retrieval is sufficient for grounding: Vector DBs reduce hallucinations by grounding the LLM, but don’t eliminate them. If retrieval fails or finds misleading context, the LLM can still err. The retrieved text must be evaluated for information quality and source trustworthiness, and the LLM may need explicit instructions to use it as authority.
Vector DB vs. traditional DB: A vector DB complements existing databases; it won’t replace them. It excels at similarity search but not structured queries or transactions. Complex systems often layer vector search with relational or graph databases, as each tool has specific strengths.
Bigger Embeddings = Better: Larger dimensionality (e.g., 1536-dim) doesn’t guarantee better results than a smaller one (e.g., 384-dim). The quality of the embedding model and its domain-specific training matters more. Bigger vectors also consume more memory and slow down search. Choose a model that balances semantic accuracy with efficiency.

To wrap up this lesson: semantic retrieval via embeddings and vector databases is a cornerstone of modern LLM systems because it addresses the fundamental limitations of keyword search. It allows our AI systems to fetch information based on meaning, which is much closer to how humans think and ask questions. But it’s not magic – understanding its mechanics and limitations helps us use it effectively. In particular, always remember the mental model here:

In other words, vector-based semantic search will find texts that sound relevant. It’s up to the system (and us as designers) to ensure those texts are used properly to produce truthful, relevant answers.

Discover more about the vector databases through our specialized course: Vector Databases: From Embeddings to Applications

Vector databases enable semantic retrieval by supplying relevant information at query time, but retrieval alone does not address how generative AI systems remember information across interactions. In the next lesson, we’ll examine memory in generative AI systems. How short-term context, retrieved knowledge, and longer-lived memory differ, and why deciding what to store, for how long, and where is a system design problem rather than a model capability.

Previous Lesson Next Lesson

Fahim Ul Haq

I’m Fahim, a software engineer specializing in generative AI, system design, and responsible AI adoption.

View Profile

Agentic AI