GraphRAG and knowledge graphs: when graphs beat plain vector search
Vector search finds chunks that are similar to the question. It struggles with questions about relationships between things ("which suppliers of our top customer also supply our competitor?"), questions that need several hops, questions about the whole collection ("what are the main themes across all incident reports?"), and exact values that must never be paraphrased. A knowledge graph stores entities and their relationships explicitly, and GraphRAG uses that structure for retrieval. It is powerful for these cases and expensive to build and maintain, so use it when your evaluation shows plain RAG failing on relationship or whole-corpus questions, not by default.
What vector search can't do well
Vector search treats your documents as a pile of independent chunks. Each chunk is matched to the question on its own.
That works for "what is the refund period for laptops?", where one chunk contains the answer. It works badly when the answer is spread across many chunks and connected through relationships:
- Multi-hop questions: "Who is the manager of the person who approved this contract?" needs one fact to find the next.
- Relationship questions: "Which of our projects use a library that has a known vulnerability?"
- Aggregation and whole-corpus questions: "What are the most common causes of outages this year?" No single chunk contains the answer.
- Entity-centred questions: "Tell me everything about Customer X", where the information is scattered across emails, contracts and tickets.
What a knowledge graph is
A knowledge graph stores information as entities (nodes) and relationships (edges).
- Entities: people, companies, products, contracts, documents, locations.
- Relationships: "works for", "supplies", "depends on", "signed", "mentions".
- Properties: attributes of entities and relationships, like dates, amounts and IDs.
"Asha manages the payments team, which owns the billing service, which depends on PostgreSQL" becomes four entities and three relationships, and a question that follows that chain becomes a graph query instead of a hope that the right chunks land together.
Graphs are stored in graph databases like Neo4j or in relational databases, and queried with query languages like Cypher or with code.
How GraphRAG works
GraphRAG is a family of approaches, and Microsoft's 2024 GraphRAG work made the term popular. The general idea:
- Extract: use an LLM to extract entities and relationships from every chunk of your documents.
- Build the graph: merge duplicate entities ("Acme", "Acme Corp", "ACME Ltd") and link everything.
- Summarise communities: group closely connected entities into communities and write a summary for each.
- Retrieve: for a question, find relevant entities and follow their relationships (local search), or use the community summaries for questions about the whole collection (global search).
- Generate: give the model the relevant subgraph, summaries and source chunks to answer from.
Microsoft's research showed this works especially well for global, sense-making questions over large collections, where plain RAG retrieves a few chunks and misses the big picture.
Exact values from structured data
There is a simpler and very practical use of structured knowledge: making sure numbers and facts are exact.
If a user asks "how much did Usha sell last quarter?", plain RAG retrieves a chunk and the model writes the number in its answer, and it can get it wrong. If the value is stored as structured data, an entity Usha with a property for sales in a given quarter, in a graph or a database, the system pulls the exact value and the model only writes the sentence around it. This is the same principle as giving the model a tool for exact values: let the data store be the source of truth for facts, and let the model handle language.
For tabular data that people query with filters and calculations, a text-to-SQL tool over the database is often simpler than a graph.
Costs and trade-offs
- Building: entity and relationship extraction needs an LLM call per chunk, which costs much more than embedding.
- Quality: extraction makes mistakes, missed entities, wrong relationships, duplicates, and they propagate into answers.
- Entity resolution: deciding that two names are the same entity is hard and important.
- Maintenance: when documents change, the graph has to be updated, and community summaries rebuilt.
- Complexity: another database, another query language, another pipeline to operate.
When to use what
| Question type | Approach |
|---|---|
| Specific facts in one place | Plain RAG with hybrid search and reranking |
| Exact numbers from structured data | Database or graph lookup through a tool |
| Multi-hop relationships | Knowledge graph queries, or agentic retrieval |
| Themes across the whole collection | GraphRAG community summaries, or map-reduce summarisation |
| Entity-centred questions | Graph of entities linked to their source chunks |
Many systems combine them: vector search for passages, a graph for relationships and exact values, and an agent that decides which to use.
From the floor
In the knowledge engine product my team builds for companies with large internal knowledge bases, one of the architectural choices is exactly the exact-values idea: important entities and values from the data are pulled into a knowledge graph, so when someone asks how much a salesperson sold last quarter, the model writes the answer conversationally, and the number comes from the stored data instead of being generated. We work with clients in sectors like defence and law enforcement, where security concerns always come first.