Hiring AI engineers: what to test for
Test whether candidates can build, deploy and explain AI systems: review a deployed project together, run an AI system design interview, give a short practical exercise with a real failure in it, and ask them to walk you through something that broke and how they fixed it. Certificates and keyword-heavy resumes tell you little.
What most interviews miss
Interviews often test definitions ("what is RAG?") that anyone can memorise, and miss whether the person can make a system work on messy data. The best single question is: show me one thing that didn't work when you built this, what you changed, and how you checked that the change solved it.
Interview loop
| Stage | Tests | Format |
|---|---|---|
| Portfolio walkthrough | Real building, depth, honesty about limits | 45 minutes on one project they choose |
| AI system design | Trade-offs, evaluation, cost, safety | 45 to 60 minutes, one worked question |
| Practical exercise | Debugging | A small RAG or agent with a planted failure, 60 to 90 minutes |
| Collaboration | Working with domain experts and product | Conversation with a product owner or domain expert |
Signals
| Strong signal | Weak signal |
|---|---|
| Talks about what broke and how they found it | Lists frameworks used |
| Has numbers from their own evals | Says "it worked well" |
| Asks about data and users first | Starts with the model |
| Knows when not to use an LLM | Adds agents to everything |
| Names limitations of their design | Defends every choice |
Build the exercise from your own system
Take a real failure from your own product (a retrieval miss, a tool error the agent ignored) and turn it into the exercise. It tests what the job needs, and candidates get a fair picture of the work.
Experience vs AI-specific skills
A senior backend engineer who has built one or two AI systems properly is often a better hire than someone with many tutorial projects, because production experience is the harder part to teach.
From the floor
My master question to builders is the same one I'd give to anyone hiring: show me one thing that didn't work when you built this, what you changed, and how you checked that the change solved it.