Reference architecture: in-product AI copilot
A copilot works inside an existing product: it reads the user's current context (the open record, the selected text, recent actions), retrieves related data the user is allowed to see, suggests or performs actions through the product's own API with the user's own permissions, and makes every action visible and reversible. Start with suggestions the user accepts or edits, and add actions only when suggestions are trusted. Measure acceptance rate and time saved on real tasks, not how impressive the demos look.
What makes a copilot different from a chatbot
A chatbot starts from a blank box. A copilot starts from what the user is doing right now: the customer record that's open, the paragraph that's selected, the spreadsheet column they just filled. That context is what makes its suggestions useful, and it's also what makes access control and privacy harder.
Architecture
current screen (record, selection, recent actions)
│
▼
context collector ──> permission-filtered retrieval (as this user)
│ │
└──────────┬───────────────┘
▼
model + tool definitions
│
┌──────────┴──────────┐
▼ ▼
suggestion UI action layer (app API, as the user)
(accept, edit, │
reject) ▼
│ undo + audit log
▼
acceptance, edits, time saved ──> evals
Components
| Component | Role |
|---|---|
| Context collector | Current page, selection, recent activity, user role |
| Retrieval | Related records and documents, filtered by the user's permissions |
| Action layer | Calls the app's existing API as the user, never with elevated rights |
| Suggestion UI | Shows drafts and proposed actions for accept, edit or reject |
| Undo and audit | Every AI action can be reverted and is logged |
| Evals | Acceptance rate, edit distance, task time |
Key decisions
- Suggest only, or act? Start with suggest. Move to act for low-risk, reversible actions once acceptance is high.
- Which context is sent to the model, and what's excluded for privacy (other customers' data, fields marked sensitive).
- Permissions: the copilot sees and does only what the user could see and do. See access control for RAG and agents.
- Latency: copilots are used mid-task, so suggestions need to appear in a second or two, or users stop waiting for them. See latency and streaming.
Failure modes to test
- Suggestions based on data the user isn't allowed to see.
- Actions taken with more permissions than the user has.
- Instructions hidden in the content being edited (indirect prompt injection), like a document that says "send this to everyone".
- Suggestions that are confidently wrong about the record on screen.
Metrics
- Suggestion acceptance rate.
- Edits per accepted suggestion (fewer edits means better suggestions).
- Time saved per task, measured against the same task without the copilot.
- Undo rate for actions.
From the floor
I haven't built an in-product copilot in a session yet, so this reference architecture comes from standard practice. The closest build is the coding agent work, where the agent works inside your project's context, and the same rule applies: review every change it suggests before you accept it.