Teardown: how an AI answer engine with citations is likely built
An answer engine likely rewrites your question into several search queries, calls one or more search indexes, fetches and cleans the top pages, chunks them, ranks the chunks for relevance, sends the best ones to a model with instructions to answer only from them with numbered citations, and checks that the citations point to real sources. It's RAG over the web, with search quality and source quality as the hard parts.
Likely pipeline
| Step | Purpose |
|---|---|
| Query understanding | Decompose and rewrite the question into search queries |
| Search | A web index or several search APIs |
| Fetch and clean | Download pages, extract the main text |
| Chunk and rank | Keep the passages that answer the question, with a reranker |
| Generate | Answer from passages with inline citations |
| Verify | Check that citations map to passages |
| Follow-ups | Suggest related questions |
Latency
Search, fetching and ranking all happen before the first token. Running searches and page fetches in parallel, caching popular pages and streaming the answer as soon as generation starts is likely how these products keep waiting time to a few seconds.
Where it fails
When search returns a joke or satire, a faithful model repeats it. That's what happened with Google's AI Overviews and the advice to put glue on pizza in 2024, which is why source quality checks matter as much as faithfulness checks. When the source itself was generated by AI and is wrong, a citation doesn't help, because the answer is faithful to a wrong source.
What you can observe yourself
- Click the citations: do they say what the answer claims?
- Ask about something very recent: does it find it, or answer from older pages?
- Ask a question with a known satirical answer online and see whether it's filtered.
From the floor
While researching the legal citation problem, I pointed out this weakness to builders: AI search tools pull from websites that look authentic, but those websites may be using LLMs to generate their blogs too, so I didn't want my data or my decisions to be based on them without checking.