RAG for regulated data: healthcare, pharma, finance and defence
In regulated industries, the question is not only whether the answers are good, but where the data goes, who can see it, and whether you can prove what happened. That usually means running every component, parsing, embedding, the vector database and the LLM, inside infrastructure the organisation controls, sometimes fully offline, plus strict access control, masking of personal data, verified citations, and an audit log of every question, retrieval and answer. Open-weight models and local tools make this possible, with trade-offs in model quality, hardware cost and the team needed to run it.
Why regulated data is different
A normal RAG system often sends document chunks to a hosted embedding API and a hosted LLM. For a hospital's patient records, a pharma company's trial data, a bank's customer files or a defence organisation's documents, that can be forbidden by law, by contract, or by the organisation's own policy.
The requirements change the architecture:
- Data residency: data may have to stay in a specific country or inside the organisation's network.
- Confidentiality: personal and sensitive data must be protected, and only the right people may see it.
- Auditability: you must be able to show who asked what, what was retrieved and what was answered.
- Accuracy and accountability: a wrong answer can have legal, financial or health consequences, so answers must be traceable to sources and reviewed by people where it matters.
Rules you will meet
Depending on the industry and country, you will hear about rules like:
- HIPAA for health information in the United States.
- GDPR for personal data of people in the European Union.
- India's Digital Personal Data Protection Act, 2023, for personal data in India.
- The EU AI Act, which sets obligations based on the risk level of an AI system.
- Sector rules from financial regulators, pharmaceutical regulators and defence authorities.
This page is not legal advice. Involve the organisation's legal and compliance team early, because their requirements decide your architecture, not the other way around.
Deployment options
| Option | Data leaves the organisation? | Typical use |
|---|---|---|
| Hosted APIs with enterprise agreements and regional processing | Yes, under contract | Less sensitive data, where the provider's terms are approved |
| Models in your own cloud account (private endpoints) | Stays in your cloud tenancy | Many enterprises, with approved cloud providers |
| On-premises | No | Hospitals, banks, government, defence |
| Air-gapped or offline | No, and no internet at all | Defence, classified data, field devices, personal privacy tools |
For on-premises and offline systems, every part has to run locally: the parser, OCR, embedding model, vector database, reranker and LLM, and even licence checks and model downloads have to be handled without internet.
Local models
Open-weight models make local RAG practical. Small and medium models, often quantized to run on a single GPU or even a laptop, can answer well from retrieved documents, because in RAG the model's job is mostly to read and summarise the context rather than recall facts.
The trade-offs:
- Quality: smaller local models are weaker at complex reasoning than the best hosted models, so test them on your real questions.
- Hardware: you buy or rent GPUs, and plan capacity for peak usage.
- Operations: you update models, patch security issues and monitor performance yourself.
See hosted vs self-hosted LLMs.
Protecting sensitive data inside the system
Even inside the organisation, not everyone should see everything.
- Access control at retrieval: filter by the user's permissions in the retrieval query. See metadata filtering and access control.
- Masking: detect personal data like names, IDs, phone numbers and account numbers, replace them with placeholders before they reach the model or the logs, and restore them only in the final interface for users allowed to see them.
- Minimisation: retrieve and send only what the question needs.
- Encryption: encrypt data at rest and in transit, including the vector database and backups.
- Retention: decide how long questions, answers and logs are kept, and delete them on schedule.
Trust and verification
In regulated settings, every answer should be traceable.
- Require citations to the specific document, page and paragraph.
- Verify in code that cited sources exist and were retrieved.
- Say clearly when the documents don't answer the question.
- Keep a person in the loop for decisions with legal, financial or clinical consequences.
- Log every question, the retrieved chunks, the answer, the model version and the user, in an audit trail that can't be edited.
Evaluation for regulated use
Beyond normal RAG evaluation, test:
- Access control: users must never receive content they aren't allowed to see.
- Masking: personal data must not appear in prompts, logs or answers where it shouldn't.
- Abstention: the system must say it doesn't know rather than guess.
- Prompt injection and data exfiltration attempts.
From the floor
Neural Vault was the first artifact we built in AI Engineer HQ: a desktop application that lets you chat with sensitive documents completely offline, an offline-first RAG system with a quantized small language model. The requirements came from real conversations with companies that don't want to share their data with any cloud provider: it had to work air-gapped, with no internet at any stage, including model loading and licence checks, use hybrid search, mask sensitive data before processing and restore it only in the interface, and link every claim to a specific paragraph in a local PDF. The users we designed it for were legal and compliance analysts, healthcare teams, the defence sector and financial risk managers.
I also gave builders a smaller example of the same idea: a homeopathic doctor who has solved around 100 cases in his own way has knowledge that exists nowhere else, and a small, private RAG system over those case notes would let him search his own experience without sending it anywhere.
Related
- Hosted vs self-hosted LLMs
- Security and governance
- Metadata filtering and access control
- Model families