Responsible AI checklist for shipping teams
Before launching an AI feature, answer ten questions in writing and have someone outside the build team review the answers. The questions cover purpose, whether AI is needed at all, the impact of wrong answers, the data, evals, fairness, transparency, human oversight, monitoring and how to switch it off. Responsible AI isn't a separate project: it's these questions asked at the right time, with written answers someone can check.
Why write it down
Most AI harm isn't caused by bad intentions. It's caused by questions nobody asked: what happens when it's wrong, who it works worse for, whether users know it's AI, who can turn it off. Writing the answers forces the questions, and a reviewer outside the team catches what the builders can't see.
Checklist
- Purpose: what problem does this solve, and for whom?
- Need: could a non-AI solution, a script, a rule or an existing product, do this well enough?
- Impact: what happens to a user when the AI is wrong, and how bad is the worst case?
- Data: what data is used, is its use allowed for this purpose, and is personal data protected?
- Evals: what's the pass rate on a realistic eval set, including hard and adversarial cases?
- Fairness: have you checked results across the user groups and languages you serve?
- Transparency: do users know they're dealing with AI, and can they reach a person?
- Oversight: which decisions or actions need human review, and who reviews them?
- Monitoring: what is sampled and reviewed after launch, and by whom?
- Off switch: who can disable it, and how fast?
How to use it
- The build team writes the answers early, during design, not the day before launch.
- Someone outside the team reviews them: security, legal, a domain expert or a product lead.
- Gaps become tasks before launch, or written, accepted risks with an owner.
- Revisit the answers when the feature changes: a new data source, a new tool, a new user group.
Fairness in practice
Fairness checks don't need a research team. Split your eval set by the groups that matter for your product (languages, regions, customer types, accents for voice) and compare pass rates. A support bot that's 95% accurate in English and 70% in Hindi isn't 95% accurate for your users.
Matching effort to risk
A tool that drafts internal meeting notes needs short answers. A system that screens job candidates, assesses credit or gives medical information needs thorough answers, and may fall under the high-risk rules of the EU AI Act. See compliance basics.
From the floor
The second question on this list, whether you need AI at all, is the one I ask most often: if a Python script, a normal workflow or an existing product solves the problem, don't add an LLM or an agent, because every AI component brings evaluation, monitoring and maintenance with it. In my webinar for business leaders, I put it this way: automations are static, a script repeats the same steps and gives you an output, so if the task is repetitive and a script can do it, use the script, and agents earn their place when the task needs decisions a fixed script can't make.
In the legal LLM project lab, the docs folder held a problem brief, the legal boundaries the tool must not cross, a risk register covering hallucination and biased answers, and the metrics, written before any code, which is this checklist in the form a project team can keep next to the code.