Data leakage and PII handling in LLM applications
Detect and mask personal data (names, emails, phone numbers, card and ID numbers) before it reaches a model or a log, keep a reverse map so you can restore it in the answer where the user should see it, set retention rules for prompts and traces, and check each provider's data use terms. Then give employees approved tools and clear rules for using AI with company data, because unapproved AI use is a bigger leak risk in most companies than any attack. Once personal data is in a third-party prompt or a log, you've lost control of where it goes.
Where data leaks
- Prompts sent to external model providers.
- Logs and traces, which often keep full prompts and outputs for months.
- Caches that return one user's answer to another.
- Fine-tuning data that teaches a model specific personal details.
- Answers that include data the user shouldn't see.
- Employees pasting company data into public AI tools.
Personal data rarely helps the model answer a question. "Update the address for Priya Sharma, priya@example.com" works just as well as "update the address for <PERSON_1>, <EMAIL_1>".
Masking with a reverse map
- Detect personal data in the input.
- Replace each value with a placeholder, and keep a map from placeholder to value for this request only.
- Send the masked text to the model.
- Restore the values in the output where the user should see them.
- Store only the masked version in logs.
The guardrail sandwich page has runnable code for this. Microsoft Presidio is an open-source option that detects many entity types with context, and supports custom recognisers for things like Aadhaar or PAN numbers.
Secrets
API keys, passwords and connection strings must never appear in prompts, system prompts or logs. Users can extract system prompts, and logs are read by more people than you think. See secrets management.
Retention
Decide how long prompts, outputs and traces are kept, and delete them on schedule. Keep the full text only as long as debugging needs it, and keep masked versions for longer-term analysis.
Provider terms
For each model provider, check:
- Whether API data is used for training (most major providers say API data isn't used by default, and you should confirm it in their current terms).
- How long they retain data, and whether zero-data-retention options exist.
- Where the data is processed, for data residency requirements.
- Whether they sign the agreements your sector needs, such as a business associate agreement for health data in the US.
You can't verify from the outside what happens to data after you send it, so the safest data is the data you don't send.
Shadow AI
Shadow AI means employees using unapproved AI tools with company data. Bans tend to push it underground. What reduces it is an approved, convenient tool with the right data protections, plus clear rules on which data can go into which tools.
In 2023, Samsung restricted employees' use of generative AI tools after reports that internal source code had been pasted into ChatGPT.
Measuring it
- PII found in logs per week (target zero). Run a PII detector over a sample of your logs, and whatever it finds is what you're currently storing.
- Share of AI use going through approved tools.
From the floor
In Neural Vault, one of the components we specified was a log sanitizer: if logs contain something like a password, it's automatically replaced with context-aware placeholder text before the logs are processed anywhere, because logs are one of the first places sensitive data leaks.
In the production pipeline I teach, the PII masker is the first guardrail in the input zone, with Presidio as the example, for two reasons: personal information is irrelevant to the model's answer, and once it's sent to a provider and stored, you can't control what happens to it.