1 · A quick read
A cheap check reads the shape of the documents: one matter or many? a roster of names, or a real story? any relationship-words at all? Rosters and multi-matter piles get flagged "unlikely to help."
Research Blog · Methodology · 2026-08-22
When to Reach for GraphRAG — and When Not To
Knowledge-graph RAG is the most over-sold idea in retrieval right now. The pitch — "turn your documents into a graph and find connections plain search can't" — is real for some document sets and a waste of money for most. We built a careful version, ran it across very different sets, and the most useful thing we can tell you is when it's worth turning on at all.
The graph builder is experimental in BDS — off by default. This note reports what we measured, and says plainly what isn't proven yet.
What We Built
The graph, not the answers.
Most graph tools extract and trust: an AI reads the text, pulls out "X relates to Y" statements, and turns them into connections. Microsoft's GraphRAG then groups related parties into clusters and summarizes each one to answer big-picture questions — it's good at those (reportedly 70–80% of the time), but building the index for a large set was famously expensive (~$33K in the 2024 version). LightRAG, the 2026 cost answer, skips the clustering and cuts that cost dramatically.
We went the other way — toward precision. Before a connection joins the graph, a separate checker has to confirm it four ways: both parties are real, the subject is the one actually acting, the stated relationship matches the text, and a specific quote backs it up. Only connections that pass all four get in. On a hand-checked sample, about 95% are correct, and each carries its source line — exactly what legal and eDiscovery work needs. The trade: we keep few connections, and all that checking is slow.
What we did — and didn't — buildWe built the graph itself, not a system that answers questions from it. It's a verified map you can open and inspect — not "answers from the graph." That part comes later.
The Result That Matters
It's the documents, not the model.
We ran the same builder on three very different document sets, and sorted every connection it found into three buckets: obvious from a single page, spanning several documents, or a real "needle" you'd never spot by reading.
| Document set | Connections | Needles | What we saw |
|---|---|---|---|
| Land-use / environmental CEQAnet — 300 docs, 57 projects | ~58 | 0 | All of them "person → their own agency" or "unit → parent department." True, verified, and useless — and 57 separate projects don't connect to each other at all. |
| One cohesive matter a single development's file | denser | few | More connected — the same parties recur within one matter — but still mostly obvious hierarchy. |
| Litigation SEC v. Ripple | 38 | ~1/3 | Same builder, same settings — now the real relationships appear: who represents whom, revolving-door SEC ties, ownership webs you'd otherwise hand-trace across dozens of filings. |
The Ripple result is modestly encouraging — nothing more.It's one set of documents and the "needle" count is our own judgment call, so it carries no statistical weight. We also haven't built the part that would use the graph to answer questions — and with connections this sparse, there may not be enough there to beat ordinary search anyway. It fits the theory; it doesn't prove it.
The lesson: a relationship graph is worth building only when the documents are one connected matter (so the same parties keep coming up) and the material actually spells out relationships in words. Rosters, notices, permits, and forms give you a verified-but-useless graph — and no better model fixes documents whose value was never about connections.
Even the case that worked had two real limits — both about the parties, not the connections. The same real party sometimes showed up as two separate entries: one official under two spellings of their name, and a scanning error that split a notable party in two. A graph that treats one person as two is quietly wrong in a way that matters. And coverage was thin: out of a set with well over a hundred thousand places where two parties are named together, only 38 connections cleared the checker. A relationship graph is only as trustworthy as the parties underneath it — so for legal work, pinning each party to one correct identity isn't a nicety, it's a requirement.
So We Gate It
Recommend it; never run it automatically.
Rather than ship a graph button that burns hours to produce an empty result, BDS checks whether a graph is even worth building first.
2 · A sample run
Runs the real pipeline on a sample and sees which kinds of relationships actually pass the checker. Only "works-for" and "part-of"? We don't recommend the graph. Ownership, representation, investigation links? We do.
3 · You decide
Building the graph is an explicit, clearly-labeled experimental step — never automatic.
Where This Leaves Us vs. the Field
GraphRAG is situational — three different bets.
Microsoft GraphRAG
Bets on big-picture summaries across a whole corpus — and pays for it in indexing cost.
LightRAG
Bets on low cost, skipping the clustering step.
BDS
Bets on trusting every connection — the right call for legal / defensible work, but only half a system until the answering layer exists.
The honest, unglamorous finding all three keep re-learning: a graph helps when the documents connect and the question spans many of them; it does nothing for a simple lookup in a pile of unrelated files. Most 2026 systems pick the method to fit the question and keep ordinary search as the default — which is exactly what BDS does.
Why We Stopped at a Verified Graph — For Now
We built the graph — and chose not to chase the answer engine.
A verified graph is one thing; using it to answer questions is another — and the second is where knowledge-graph projects quietly swallow unbounded effort.
Building a graph-answering layer isn't a task, it's a program: build the machinery to search the graph, write a question set with an answer key, run a controlled head-to-head against the search we already have, then repeat it on a second set of documents to prove it generalizes. And on our own evidence, the likely payoff is narrow — a modest edge on a specific kind of connect-the-dots-across-the-whole-matter question, or nothing at all.
We looked hard at the cheap path — LightRAG (runs fully local, open-licensed, and skips the expensive clustering step). It clears the engineering bar. But by its own published benchmarks it lands at about the same accuracy as Microsoft's GraphRAG — its advantage is cost, not accuracy. So the honest read is a large, speculative investment for a marginal expected return.
What's proven stays; what isn't, we don't claim.The verified graph is proven — precise, evidence-cited, defensible. Graph-driven answering is not; it's an unproven path. With finite resources and a production-ready core, we chose not to chase it — a resource-allocation decision, not a verdict that it can't work.
What would change our mind: a concrete need for cross-document questions our current search demonstrably can't answer, or a controlled test showing graph-answering wins after you count the cost. We've already scoped that experiment — LightRAG-style search over our own verified connections, with stop criteria set in advance. If the need shows up, we run it.
Until then, our relationship graph stays what it honestly is: a verified, source-backed map you can trust — an asset, not an answer engine.
Provenance
Grounded in BDS GraphRAG runs on land-use (CEQAnet), litigation (SEC v. Ripple), and safety-investigation (NTSB) document sets, August 2026. The ~95% precision is hand-checked on a small sample; the Ripple "needle" share is our own judgment call on a single set, with no graph-vs-search test yet — an unproven path, reported as such. The Microsoft GraphRAG and LightRAG figures are from public sources, cited as reported, not benchmarked by us.
— V.I. lab notes, 2026-08-22