Reference architecture: enterprise knowledge engine (on-prem)
An enterprise knowledge engine connects to the company's document stores and databases, ingests content with its permissions, indexes it for hybrid search (and optionally a knowledge graph of entities and relationships), answers questions with citations using models hosted inside the company's network, and keeps an audit trail of who asked what and which sources were used. Exact values like sales figures come from structured queries, not from generation. For regulated or security-sensitive organisations, the whole thing can run on-premises with no data leaving the network.
Who needs this
Companies with years of knowledge spread across drives, wikis, tickets, CRMs and databases, where people spend hours finding answers that already exist somewhere. For many of them, sending that data to an external AI service isn't allowed: defence, law enforcement, healthcare and finance often run their own servers, and some don't connect them to the internet at all.
Architecture
sources: drives, wikis, tickets, CRM, databases
│
▼
connectors ──> ingestion (parse, chunk, enrich, embed, sync permissions, handle deletes)
│
┌────────────┼──────────────┐
▼ ▼ ▼
vector index keyword index entity graph
└────────────┼──────────────┘
▼
question ──> rewrite ──> hybrid retrieval (permission filters) ──> rerank
│ │
└── structured questions ──> SQL tools ───────┤
▼
local model (inside the network) ──> answer with citations
│
audit log, expert verification, traces
Components
| Component | Role |
|---|---|
| Connectors | Drives, wikis, ticketing, CRM, databases |
| Ingestion | Parse, chunk, enrich, embed, sync permissions, handle deletes |
| Indexes | Vector plus keyword, optional entity graph |
| Query path | Rewrite, hybrid retrieval with permission filters, rerank |
| Structured data tools | SQL over databases for exact values |
| Models | Open-weight models on internal GPUs, small models for routing and guardrails |
| Verification | Citations, and a view where experts can check and correct the graph behind an answer |
| Audit and observability | Every query, user, sources and answer logged |
Key decisions
- On-premises or private cloud, driven by the organisation's policy, not by preference.
- How permissions are mirrored from the source systems and kept in sync, including when someone leaves the company.
- Which questions go to retrieval and which to structured queries. See exact values.
- Which open-weight models fit the hardware, at which quantization. See run LLMs locally.
- How deletes propagate, so removed documents stop appearing in answers.
Failure modes to test
- An answer using a document the user isn't allowed to see.
- A number in the answer that doesn't match the source.
- An answer from an outdated version of a document.
- Accuracy dropping as conversations get longer and more context accumulates.
Metrics
- Answer acceptance by subject matter experts.
- Retrieval recall on a labelled question set.
- Citation coverage (share of claims with a supporting source).
- Time to find information compared with the old process.
From the floor
At Dextar, we're building a knowledge engine for company data, focused on on-premises deployments with open-source models, because some of the organisations interested in it, including in defence and law enforcement, run their own servers and don't use the internet for this kind of work. The design pulls the important entities from the data into a knowledge graph, so that exact values, like figures from sales data, come from the data, and the LLM turns them into a conversational answer.
Neural Vault, one of the AI Engineer HQ artifacts, explored the same idea with local LLMs: fully offline, with hybrid search, citations and PII masking, so a company's documents never leave its machines.