AI system design interview: keep session memory for 10,000 returning users
Keep active sessions in a fast store like Redis, with a buffer of recent turns and a TTL. When a session ends or goes idle, write a summary and the full transcript to durable storage keyed by user, and extract long-term facts into a memory store. When the user returns, load the last summary and retrieve only the past turns or facts relevant to the new question, within a fixed context budget, and key everything by user and tenant so context never bleeds between users.
Question
10,000 users chat with your assistant and come back a week later expecting it to remember their last conversation. How do you manage session memory without context bleeding between users?
Clarify
- How many are active at the same time.
- How long a session lasts and how many turns it has.
- What "remember" means: show the last conversation, continue it, or use facts from it?
- Retention and deletion requirements.
Estimate
If 10% of 10,000 users are active at peak, that's 1,000 hot sessions. A buffer of 20 turns at about 500 tokens each is around 10,000 tokens, roughly 40 KB of text per session, so 1,000 sessions need about 40 MB in the hot store, which is small. The real cost is in tokens: loading too much history into every request makes each call slower and more expensive.
Architecture
| Tier | Holds | Store |
|---|---|---|
| Hot | Recent turns of active sessions | Redis, key tenant:user:session, TTL |
| Warm | Session summaries and full transcripts | Database keyed by user |
| Long-term | Durable facts and preferences | Memory store with retrieval |
On return: load the last summary, retrieve relevant past turns or facts for the new message, and assemble the prompt within a budget, for example 2,000 tokens of memory.
Deep dive: context bleeding
Bleeding usually comes from shared state: a global conversation object in the app server, a cache key without the user ID, or a memory search without a user filter. Keep the request path stateless and load state by user ID on every request, include user and tenant in every key and filter, and test with concurrent sessions that contain recognisable, different content.
Failure modes
| Failure | Mitigation |
|---|---|
| Context bleeding between users | Stateless servers, scoped keys, concurrency tests |
| Prompts growing with every turn | Buffer window plus summaries, fixed budget |
| Lost sessions on restart | Persist summaries on idle, not only on explicit end |
| Old irrelevant context confuses answers | Retrieve by relevance, not by recency only |
What interviewers listen for
Separate hot and durable tiers, a context budget, and isolation designed and tested.
From the floor
A builder asked exactly this in my inference pipeline lecture, including the context bleeding they'd seen in production. I said in the session that I didn't have a direct answer ready, and listed where it starts: the cache with a buffer window for active users and reclaiming that memory, long-term semantic memory in a store like Mem0, and the inference zone, because hardware constraints apply, and I said I'd work on the problem and come back with a proper response, which is this page.