AI system design interview: design long-term memory for an assistant
Define a write policy first: after each session, a model extracts candidate facts, and only durable ones (preferences, ongoing projects, stated constraints) are kept, each with its source and a last-confirmed date, and sensitive categories are never stored without consent. At query time, retrieve only facts relevant to the current message, update or expire facts when the user contradicts them, and let users see, edit and delete their memories.
Question
Design memory so an assistant remembers users' preferences and past context across sessions.
Clarify
- What should be remembered: preferences, facts about the user, past tasks?
- Privacy rules and consent.
- Whether users expect to see and control what's remembered.
Memory types
| Type | Example | Storage |
|---|---|---|
| Working memory | The current conversation | Context window, session store |
| Episodic | "Last week we drafted the Q3 report" | Summaries of past sessions |
| Semantic | "Prefers answers in bullet points" | Fact store with metadata |
| Procedural | "Always check the style guide first" | Instructions or rules |
Architecture
1 - Extract: after a session, a model proposes candidate facts with the turn they came from. 2 - Filter: keep durable facts, drop one-off details and blocked categories. 3 - Store: each fact with user ID, source, created and last-confirmed dates. 4 - Retrieve: at query time, search facts relevant to the current message, within a token budget. 5 - Update: when a new fact contradicts an old one, replace it and keep history. 6 - Control: a page where users see, edit and delete what's stored.
Deep dive: staleness and contradictions
"Works at Company A" stops being true. Store when each fact was last confirmed, prefer recent facts when two conflict, and expire facts that haven't been confirmed for a long time, depending on the type.
Failure modes
| Failure | Mitigation |
|---|---|
| Remembering something sensitive | Blocked categories in the write policy, consent |
| Stale facts used confidently | Last-confirmed dates, expiry |
| Irrelevant memories injected into every answer | Retrieve by relevance, small budget |
| One user's memory used for another | Keyed by user and tenant, tested |
Evals
Conversations where memory should be used and where it shouldn't, scored for both, plus how often users correct or delete memories in production.
What interviewers listen for
A write policy, staleness handling, user control and privacy.
From the floor
When a builder asked about session memory for returning users in the inference pipeline lecture, one of the places I said the answer starts is long-term memory in a dedicated store like Mem0, separate from the short buffer of recent turns kept in the cache.