AI system design interview: deploy an LLM on-prem with no internet
Everything runs locally, including the small models people forget: the LLM, embeddings, reranker and guardrail models. Pick open-weight models that fit the available GPUs (quantized if needed), serve them with an engine like vLLM, mirror every model, package and container into an internal registry, block downloads at runtime, and bring updates in through a reviewed process with an eval run before switching. Size the hardware from users, context length and the KV cache.
Question
A government agency wants document Q&A on its own servers with no internet access.
Clarify
- Available hardware: GPU models, count and memory.
- Number of users and peak concurrency.
- Document volume, languages and formats.
- Security requirements: classification levels, audit, who may update the system and how.
Estimate
An 8B model at 16-bit needs about 16 GB for weights, and a 70B model about 140 GB, or about 35 GB at 4-bit. Add the KV cache: about 0.13 MB per token for a Llama 3 8B-shaped model, so 50 concurrent users at 8,000 tokens need about 52 GB more. That's the arithmetic that decides between a smaller model, quantization, or more GPUs. See KV cache.
Architecture
| Layer | Component |
|---|---|
| Models | Open-weight LLM, embedding model, reranker, guardrail classifiers, all local |
| Serving | vLLM or a similar engine, behind an internal gateway |
| Retrieval | Local vector and keyword indexes, permission-aware |
| Ingestion | Local parsing and OCR |
| Supply chain | Internal registry for models, packages and containers, checksums verified |
| Observability | Local tracing and audit logs |
| Updates | New versions brought in on approved media, scanned, evaluated, then promoted |
Deep dive: updates without internet
Nothing should try to reach the internet at runtime, so set libraries to offline mode and block egress at the network level. New model versions, packages and documents arrive through an approved path, are scanned and checksummed, and pass the eval suite in a staging environment before the switch.
Failure modes
| Failure | Mitigation |
|---|---|
| A library tries to download at runtime | Offline mode, blocked egress, tested before delivery |
| Model too large for the GPUs | Quantization, smaller model, measured against evals |
| Capacity runs out at peak | Size from concurrency and context, queue with limits |
| Untraceable changes | Audit log for every model, prompt and config change |
What interviewers listen for
Everything local including the small models, a supply chain process for updates, and capacity numbers.
From the floor
We have worked with government and law enforcement organisations where project servers have no internet access, so this question comes from real constraints.
In the Neural Vault build in AI Engineer HQ, we designed for professionals who can't trust a cloud provider with their files: the application runs air-gapped, logs pass through a sanitiser, and updates arrive through an update agent that reads an encrypted USB drive instead of the internet.