How to Build GraphRAG for Enterprise Knowledge
How to build GraphRAG: when entity graphs beat chunks, extraction, community summaries, hybrid retrieval, and evals against a document-RAG baseline in 2026.
To build GraphRAG for enterprise knowledge, extract entities and relations from your documents into a graph, generate community summaries for global questions, and retrieve with a hybrid of vector, keyword, and graph traversal — then keep the system only if it beats chunk RAG on multi-hop evals. Entity-graph retrieval is an addition to hybrid search, not a replacement for citations, ACLs, or a golden set.
What you’ll build
- A documented decision that GraphRAG beats chunk RAG on a named question class
- An extraction pipeline that emits entities, relations, and source spans into a graph the client owns
- Community summaries for global questions, with citations back to source chunks
- Hybrid retrieval: keyword + vector + graph traversal, merged and re-ranked at query time
- Eval deltas vs the chunk-RAG baseline on hit rate, faithfulness, and multi-hop accuracy
Before you start
- 01A corpus where relationships matter (orgs, people, systems, clauses) and a document RAG baseline you can score
- 02An identity map or canonical names for at least the high-value entity types
- 03A graph store (or Postgres with edges) plus the existing vector and BM25 indexes
- 04A golden set that includes multi-hop and “who is connected to whom” questions
- 05Query-time ACL metadata on source documents that can flow to entities and communities
Key takeaways
- 01
GraphRAG helps when the question needs relations or a corpus-level summary; it is overhead for single-chunk FAQs.
- 02
Extraction quality and name canonicalization dominate; a messy graph is worse than no graph.
- 03
Community summaries need source spans or they hallucinate at cluster scale.
- 04
Hybrid retrieval (BM25 + embeddings + traversal/re-rank) should remain the query path.
- 05
Query-time ACLs must apply to nodes, edges, and summaries derived from documents the user cannot read.
When graphs beat chunks
Chunk RAG is the default for “what does the travel policy say?” GraphRAG is for questions whose answer is a path or a theme: “which systems does vendor X connect to that are in scope for the new control?” or “what are the recurring failure modes across last quarter’s incident reports?” Entity graph retrieval exists to walk those links instead of hoping two facts landed in the same 500-token window.
If you cannot name 20 questions of that shape, do not start a graph project. Knowledge graph RAG is expensive to extract and easy to get wrong. The honest sequence is baseline, fail, then graph — not graph as a brand.
Extraction, canonicalization, and the ontology
Keep the ontology small. Extraction is an LLM (or mixed rules+LLM) over structure-aware chunks that must return JSON: entities, types, relations, and source spans. Canonicalize against existing systems of record. Merge nodes on shared identifiers (email, SKU, ticket key) before you merge on string similarity.
Every edge stores document id, chunk id, span, time, and ACL. Incremental extract on change events; delete edges when the source chunk is gone. Unbounded graphs from a one-shot crawl of the whole Drive will include drafts and gossip. Start with the two sources that already matter to the baseline RAG.
Community summaries for global questions
Community detection groups related nodes so you can summarize a theme (“payments incidents”) without stuffing the whole corpus into a context window. The Microsoft GraphRAG pattern — local communities, hierarchical roll-ups, map-reduce summaries — is a reasonable default. The enterprise add-ons are citations and ACLs on those summaries.
Generate summaries only from member spans. Score faithfulness. If a summary cannot cite, it cannot be retrieved. Refresh summaries when membership changes. Global search should retrieve a handful of community summaries; local search should traverse or return chunks. Mixing both without a router produces long, vague answers.
Hybrid retrieval plus graph traversal
Query path: detect entities in the question, hybrid-search chunks (vector + BM25), traverse bounded hops from matched nodes, optionally pull community summaries for global intent, then re-rank the union. Reciprocal rank fusion is enough to start. Cap hops (usually 1–3) and cap nodes so a hub person does not pull the entire company.
Generation is still grounded generation: answer from retrieved chunks and cited summaries, refuse when empty. The graph does not license the model to invent an edge. If traversal returns nodes the user cannot read, drop them before the prompt — same existence-leak rule as document RAG.
Evals, ACLs, and operations
Compare to the chunk baseline on the same gold: multi-hop accuracy, hit rate, faithfulness, p95 latency, and extract cost. A graph that wins multi-hop and loses single-hop should be routed, not used universally. Pin extractor, community job, and generator versions. Extraction drift is a silent graph poison.
A four-week implementation is tight: ontology and gold in week 1, extract on a narrow corpus in week 2, hybrid+graph query and shadow in week 3, evals and runbooks in week 4. Wider ontologies need more time. The client owns the graph, datasets, evals, and runbooks. 30 days on-call after handover is for the jobs that re-extract and re-summarize, not for a mystery cluster.
- Baseline first: GraphRAG only on questions chunk RAG fails
- Small ontology, canonical names, spans and ACLs on every edge
- Community summaries with citations, not ungrounded blurbs
- Query: BM25 + vectors + bounded traversal, then ACL filter
When not to build GraphRAG
Do not build it for a FAQ bot, a single SharePoint site of policies, or a team that cannot staff extraction quality. Do not build it as a replacement for a governed catalog or a data warehouse — those already are graphs of a different kind. Text-to-SQL remains the right tool for metric questions on tables.
If ACLs cannot flow to nodes, stop. A graph that ignores permissions is an intranet-wide leak with nicer diagrams.
Step-by-step build
- 1
Prove the question class on a baseline
Run your existing chunk RAG (see /how-to/build-rag-ai-agent-organization-knowledge and /how-to/evaluate-rag-system) on a golden set that includes multi-hop and “summarize this theme” items. If hit rate is already high on those, stop. If failures are missing relations or drowned local chunks, you have a GraphRAG candidate.
- 2
Define entity and relation types
Write a small ontology: 5–15 types (Person, System, Policy, Vendor) and allowed relations. Bigger ontologies fail extraction. Attach each type to a canonical source when you have one (HRIS, CMDB, legal entity list).
- 3
Extract entities, relations, and spans
Chunk as you already do, then run an extractor that emits typed nodes, edges, and character spans. Canonicalize names. Write to a graph store with document id, chunk id, and ACL metadata on every edge. Re-extract on document change; do not leave stale edges.
- 4
Build communities and summaries
Cluster the graph (hierarchical communities). Summarize each community with an LLM that may use only member spans, and store citations. Global questions retrieve summaries first; local questions traverse or fetch chunks. Drop summaries that fail a faithfulness check against their spans.
- 5
Query with hybrid + graph
At question time: hybrid-search chunks, match entities in the query, traverse a bounded number of hops, and fetch community summaries when the question is global. Merge with reciprocal rank fusion and a re-ranker. Filter every candidate by the user’s live ACLs before generation.
- 6
Evaluate against the chunk baseline
Score retrieval hit rate, multi-hop accuracy, faithfulness, and latency/cost vs the baseline on the same golden set. Shadow on live traffic. Keep GraphRAG only on the routes it wins; send single-hop lookups back to chunk RAG. Hand over graph, evals, and runbooks to the client.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Building a graph because the paper was interesting
If questions are single-span lookups, chunk RAG with hybrid retrieval wins on cost and ops. GraphRAG pays for itself on multi-hop and global summaries — prove that on a golden set first.
Extraction without canonicalization
“Acme,” “ACME Inc,” and “Acme, Inc.” as three nodes make traversal lie. Resolve entities against a dictionary or IdP before you write edges.
Dropping hybrid search
Graphs are weak on exact error codes and strong on relations. Keep BM25 + vectors and add traversal; do not replace the retriever you already measured.
Community summaries that cannot cite
A fluent cluster summary without source chunk IDs is a new hallucination layer. Store supporting spans on every summary and refuse to use summaries that lack them.
ACLs only on documents, not on graph hits
An entity node connected to a restricted HR doc can leak existence or facts. Filter edges and communities at query time using the asking user’s document ACLs.