Vector index internals: build a vector store in pure Python, then HNSW, IVF and PQ
At its core, a vector store compares a query vector with stored vectors and returns the closest ones. Doing that against every vector (brute force) is exact and simple, and its cost grows with every vector you add. Approximate nearest neighbour (ANN) indexes make search fast at scale by checking only a small part of the collection: HNSW navigates a graph of neighbours, IVF searches only the closest clusters, and product quantization compresses vectors so more fit in memory. They trade a little recall for a lot of speed, and their settings control that trade.
Build one first
The best way to understand a vector database is to build the simplest one. You need three functions, a dot product, a magnitude and cosine similarity, and a list to store vectors in.
import math
def dot(a: list[float], b: list[float]) -> float:
return sum(x * y for x, y in zip(a, b))
def magnitude(v: list[float]) -> float:
return math.sqrt(dot(v, v))
def cosine(a: list[float], b: list[float]) -> float:
return dot(a, b) / (magnitude(a) * magnitude(b))
class VectorStore:
def __init__(self) -> None:
self.items: list[tuple[str, list[float], dict]] = []
def add(self, text: str, vector: list[float], metadata: dict | None = None) -> None:
self.items.append((text, vector, metadata or {}))
def search(self, query: list[float], k: int = 3, where: dict | None = None) -> list[tuple[float, str]]:
candidates = [
(cosine(query, vec), text)
for text, vec, meta in self.items
if not where or all(meta.get(key) == value for key, value in where.items())
]
return sorted(candidates, reverse=True)[:k]
store = VectorStore()
store.add("the cat sat on the mat", [0.9, 0.1, 0.0], {"topic": "pets"})
store.add("a kitten on the sofa", [0.8, 0.2, 0.1], {"topic": "pets"})
store.add("the stock market fell today", [0.0, 0.1, 0.9], {"topic": "finance"})
print(store.search([0.85, 0.15, 0.05], k=2))
print(store.search([0.85, 0.15, 0.05], k=2, where={"topic": "finance"}))
That is a working vector store with metadata filtering. In a real build, the vectors come from an embedding model instead of being written by hand, and the math page explains each function.
Why brute force stops working
The search above compares the query with every stored vector. That is called exact or brute-force search, and its cost grows linearly: twice the vectors, twice the time.
With NumPy and normalised vectors, brute force over 100000 vectors of 384 dimensions takes milliseconds, so for many projects it is genuinely enough. At tens of millions of vectors, with many queries per second, it becomes too slow and too expensive, and that is where approximate indexes come in.
Approximate nearest neighbour search
ANN indexes don't guarantee finding the exact closest vectors. They find vectors that are very likely among the closest, by searching only a small part of the collection.
The quality measure is recall: of the true top 10 neighbours, how many did the index return? Good indexes reach recall above 95% while checking only a tiny fraction of the vectors.
HNSW: a graph of neighbours
HNSW (Hierarchical Navigable Small World) is the most widely used index in vector databases.
It connects each vector to a few of its nearest neighbours, forming a graph. Then it builds several layers: the top layer has a few vectors with long-range links, and each lower layer has more vectors and shorter links, down to the bottom layer that contains everything.
A search starts at the top layer, jumps quickly to the area of the space near the query, then moves down layer by layer, refining the search among closer and closer neighbours. It works like finding an address: country, then city, then street, then house.
Key settings:
M: how many neighbours each vector links to. Higher means better recall and more memory.ef_construction: how carefully the graph is built. Higher means a better graph and slower indexing.ef_search: how many candidates are explored per query. Higher means better recall and slower queries, and you can tune it per query.
HNSW gives fast, high-recall search and handles inserts well, at the cost of memory, because the graph is usually kept in RAM.
IVF: search only the nearest clusters
IVF (inverted file index) first groups all vectors into clusters, using k-means, each with a centre point. At query time, it finds the few clusters whose centres are closest to the query and searches only inside them.
Key settings:
nlist: the number of clusters.nprobe: how many clusters to search per query. Higher means better recall and slower search.
IVF uses less memory than HNSW and builds fast, and it can miss neighbours that sit just across a cluster boundary, which is why nprobe matters.
It needs to be trained on a sample of your data, and retrained if the data changes a lot.
Product quantization: compress the vectors
A 1536-dimension vector in 32-bit floats takes about 6 KB. A billion of them doesn't fit in memory.
Product quantization (PQ) splits each vector into small sub-vectors and replaces each sub-vector with the ID of the closest entry in a small learned codebook. A vector of thousands of bytes becomes a few dozen bytes, and distances can be estimated directly from the codes.
The cost is accuracy, so PQ is often combined with IVF (IVF-PQ) and followed by a re-scoring step that recomputes exact distances for the top candidates. Simpler scalar quantization, storing each value as an 8-bit integer instead of a 32-bit float, cuts memory by four with a small loss, and many databases support it.
Choosing an index
| Index | Strength | Weakness | Good for |
|---|---|---|---|
| Flat (brute force) | Exact, simple | Slow at large scale | Up to hundreds of thousands of vectors |
| HNSW | Fast, high recall, good with updates | Memory-hungry | Most production RAG |
| IVF | Lower memory, fast to build | Needs training, boundary misses | Large, mostly static collections |
| IVF-PQ | Very low memory | Lower recall without re-scoring | Very large collections |
Most vector databases choose HNSW by default, and that is the right choice for most RAG systems.
Filtering and ANN
Filters, like "only documents this user can see", interact with ANN indexes in tricky ways. If you search the index first and filter afterwards, a strict filter can leave you with fewer than k results, or none. If you filter first, the index may not help. Good vector databases handle this inside the index search, and it is worth testing your most restrictive filters at realistic scale.
Measure recall on your data
Index defaults are tuned for general benchmarks.
To check your setup, compute the exact top 10 with brute force for a sample of queries, run the same queries through your index, and measure what share of the exact results it found.
Then adjust ef_search or nprobe until recall and latency are both acceptable.
From the floor
To show builders what is inside a vector database, I built one in about an hour in plain Python during a session: embedding sentences with a free Hugging Face model, writing the dot product, magnitude and cosine similarity functions by hand, and searching with them. We showed that "cat", "kitten" and "sofa" end up close together, and "stock market" far away, and that once you have built this, a vector database stops being a black box.
On Build30 day 5, the build was an in-memory vector store, the same idea as the code on this page, so builders could see exactly what a vector database adds on top of it.