Embeddings: what they are and what they are not
An embedding is a vector, a list of hundreds or thousands of numbers, that represents the meaning of a piece of text, so that texts with similar meanings get vectors that point in similar directions. You compare embeddings with cosine similarity to find related text, which is how semantic search and RAG retrieval work. Embeddings measure similarity, not truth or relevance: they can't tell you whether a document answers a question, they struggle with exact codes, numbers and negation, and they only work with vectors from the same model.
Why meaning needs numbers
How does an AI know that "king" is closer to "queen" than to "banana"? It doesn't understand words the way you do. It understands locations.
Every word, or every sentence, is turned into coordinates in a space with many dimensions, and the training process places things with similar meanings near each other. "Kitten" and "puppy" end up on the same shelf, close together, and "car" ends up somewhere else entirely. When an AI seems to grasp the meaning of your text, it is doing a very fast match to see which items are neighbours in that huge space. It is a brilliant librarian who knows exactly where everything is shelved, not a brilliant reader.
You might ask why we don't just give every word a number, apple is 1, banana is 2, and so on. The problem is that those numbers carry no meaning. Nothing about 1 and 2 tells a machine that apple and banana are both fruit, or that "Python" the language is unrelated to "python" the snake, so a single number per word can't capture context or intent.
Before embeddings: counting words
The early approach to turning text into numbers was to count words. For "I like oranges. Do you like oranges?", a bag-of-words vector counts each word: "like" appears twice, "oranges" twice, and so on.
It is simple and still useful for some tasks, but it ignores word order and meaning entirely. "Dog bites man" and "man bites dog" get the same vector, and "car" and "automobile" are as different as "car" and "banana".
Word embeddings
In 2013, a team at Google released Word2Vec, which learned a vector for each word by looking at which words appear near it in a large amount of text. In 2014, researchers at Stanford released GloVe, which learned vectors from how often words appear together across a whole corpus.
Both produced vectors where meaning shows up as geometry. Similar words cluster together: refrigerator, oven and microwave in one area, charger and battery in another. Relationships even become directions, so the vector from "man" to "woman" is roughly the same as the vector from "king" to "queen".
Their limit is that each word gets exactly one vector. "Apple" has the same vector whether you mean the fruit or the company, and "bank" has the same vector in "river bank" and "bank account".
Contextual embeddings
Transformer models fixed that. BERT, released by Google in 2018, produces a representation for each word that depends on the whole sentence around it, because the attention layers mix in information from the other words. In "I sat on the river bank" and "I deposited money at the bank", the two "bank" tokens end up with different vectors.
Today's embedding models are built on the same idea, and they usually produce one vector for a whole sentence, paragraph or document, by combining the token representations. Open models like the sentence-transformers family on Hugging Face, for example all-MiniLM-L6-v2 with 384 dimensions, and embedding APIs from OpenAI, Google, Cohere, Voyage and others, all work this way.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
sentences = [
"How do I reset my password?",
"I forgot my login credentials.",
"What is the refund policy for damaged items?",
]
vectors = model.encode(sentences, normalize_embeddings=True)
print(vectors.shape)
print(vectors @ vectors.T)
The first two sentences share almost no words, but their similarity score is high, because they mean nearly the same thing. The third one is about something else, and its score with the other two is low.
Measuring similarity
Once texts are vectors, comparing meaning becomes geometry. There are three common measures.
- Cosine similarity compares the direction of two vectors, ignoring their length. It ranges from -1 to 1, and higher means more similar. This is the most common measure for text.
- Dot product multiplies the vectors position by position and adds the results. If the vectors are normalised to length 1, the dot product is exactly the cosine similarity, which is why many systems normalise embeddings and use the faster dot product.
- Euclidean distance is the straight-line distance between the two points. Smaller means more similar. For normalised vectors, it ranks results the same way as cosine similarity, because the two are directly related.
Use the measure your embedding model was trained for, which its documentation states. The math page shows how to calculate cosine similarity from scratch, with your own dot product and magnitude functions.
Keyword search and semantic search
Keyword search matches the exact words. Search for "data" and you get documents containing "data". It is fast and precise, but it fails when the user's words differ from the document's words, and some keyword systems fail even on a spelling mistake.
Semantic search compares meaning. A user who types "my parcel never arrived" finds the document titled "Missing shipments", because their embeddings are close, even with no words in common. That understanding of context and intent is why chatbots built on embeddings understand questions so much better than old keyword-based bots.
Context matters in the other direction too. "Your t-shirt is killer" is a compliment, even though "killer" on its own sounds negative. A system that counts positive and negative words gets it wrong, while a model that represents the whole sentence gets it right.
What embeddings are used for
- Semantic search and RAG: embed every chunk of your documents, embed the user's question, and retrieve the closest chunks.
- Clustering: group support tickets, reviews or documents by topic without labelling them first.
- Classification: use embeddings as features for a simple classifier, often with very little labelled data.
- Deduplication: find near-duplicate documents or questions.
- Recommendations: find items similar to the ones a user liked.
- Anomaly detection: find texts that are far from everything else.
What embeddings are not
This is where most retrieval problems come from, so it is worth being precise.
Similar is not relevant
Embeddings tell you that two texts are about similar things. They don't tell you whether a document answers a question. "What is the refund period for electronics?" is very similar to a paragraph that says "Refund periods vary by category", which doesn't contain the answer. That is why good RAG systems add a reranking step that reads the question and each candidate together.
Similar is not true
An embedding encodes what a text is about, not whether it is correct. A wrong statement and a correct statement on the same topic can be very close in embedding space.
Exact identifiers get blurred
Product codes, error codes, invoice numbers, names and version numbers are often handled poorly, because embeddings capture general meaning, and "ERR-4012" and "ERR-4021" look almost identical in meaning. For these, keyword search is better, which is why most production systems use hybrid search, combining both.
Negation and small differences
"The patient has diabetes" and "The patient does not have diabetes" are about the same topic, so their embeddings are close, even though they mean opposite things. Embeddings are not designed to capture these differences reliably.
Long text gets diluted
One vector for a 30 page document has to represent everything in it, so specific details get lost. That is why documents are split into chunks before embedding, and why chunk size matters.
Vectors from different models don't mix
Each embedding model creates its own space. A vector from one model can't be compared with a vector from another, even if they have the same number of dimensions. If you change your embedding model, you have to embed all your documents again.
Storing embeddings
An embedding is usually stored as 32-bit floating point numbers, 4 bytes per dimension. A 1536-dimension vector is about 6 KB, so a million chunks is about 6 GB of vectors before any index.
Ways to reduce that:
- Choose a model with fewer dimensions if it performs well enough on your data.
- Some models support shortening their vectors (Matryoshka embeddings), keeping only the first part with a small loss of quality.
- Quantize the vectors to 8-bit integers or even single bits, which many vector databases support.
For small projects, a NumPy array is enough. For anything bigger, use a vector database, which stores the vectors with an index for fast nearest neighbour search and lets you filter by metadata.
Choosing an embedding model
The MTEB leaderboard on Hugging Face compares embedding models across many tasks and languages, and it is a good place to build a shortlist. But the ranking on general benchmarks doesn't tell you how a model performs on your documents and your users' questions.
Check these for your use case:
- Quality on your data: build a small test set of real questions with the chunks that should be retrieved, and measure how often each model finds them.
- Languages: if your users write in more than one language, test those languages specifically.
- Maximum input length: text longer than the limit is cut off silently.
- Dimensions: more dimensions can mean better quality, but also more storage and slower search.
- Cost and hosting: API price per million tokens, or the hardware to run an open model yourself, and whether your data is allowed to leave your infrastructure.
Embedding model selection covers this in detail.
From the floor
To show builders what is inside a vector database, we built one in about an hour in plain Python: embedding sentences with a free model from Hugging Face, writing our own dot product, magnitude and cosine similarity functions, and searching with them. Once you have written those three functions yourself, a vector database stops being a black box.
In a workshop on how LLMs work, a builder asked whether "vector representation" and "word embeddings" are the same thing, and they are: an embedding is the vector representation of a word or a piece of text. The follow-up question was whether ChatGPT uses Word2Vec, and it doesn't, it works with subword tokens and learns its own embeddings as part of the model.