AI production readiness checklist
A short checklist for teams that already have an LLM, RAG or agent feature working in a demo and need to decide whether it can face real users. If several answers are unclear, the feature is not blocked; it is unmeasured.
The twelve questions
- Can you reproduce any answer the system gave last week, including sources, prompt, model and retrieval index version?
- Do you have a fixed evaluation set that runs before a prompt, model or retrieval change ships?
- Can a user receive an answer built from a document they are not allowed to open?
- Do you know cost per request, per customer and per feature, including worst-case inputs?
- What does the product do when the model is slow, unavailable, unsure or missing evidence?
- Can an agent perform the same write action twice after a timeout or retry?
- Are destructive or high-impact actions bounded by confirmation, limits and attribution?
- Are hallucinations tracked as product failures, not support anecdotes?
- Can support or engineering inspect what was retrieved and why for a disputed answer?
- Does every output record model, prompt, index and configuration versions?
- Are logs retained long enough to answer enterprise, audit or incident questions?
- Is there a clear owner for quality, cost and safety regressions after launch?
How to read the answers
Mostly yes
The feature has the basic shape of an operable AI system. The next work is usually hardening, cost control and better evaluation coverage.
Mixed answers
This is the common case. The feature can move forward, but the unclear answers should become explicit backlog before launch.
Mostly no
The demo may be useful, but production risk is still invisible. Ship only behind narrow limits, or review the architecture first.
Unknown
Unknown is its own risk. If nobody can answer a question, the system probably does not record enough evidence yet.
Where teams usually discover gaps
Retrieval
The answer is stored, but not the exact sources that produced it. Re-running the same question against a changed index proves little.
Permissions
Access checks happen after retrieval, or only at the UI layer. The model may have already seen data the user should not access.
Evaluation
There is no stable benchmark, so prompt improvements are judged by a few examples that happened to be tested that day.
Operations
Latency, cost and failure behavior are not treated as product requirements until a customer finds the edge case.
If you need a review
- Pick the AI workflow where being wrong is most expensive
- Send a few lines about the system, data sources, users and deadline
- We tell you whether a 10-business-day readiness review fits
- If it fits, the output is a written report with risks, evidence gaps and a 30-day plan
Questions this raises
Is this checklist enough by itself?
No. It is a way to notice risk before it becomes an incident. The detailed answer depends on the implementation and the business context.
Does every AI feature need all of this?
No. Internal helpers and low-risk features need less machinery than customer-facing agents or systems touching money, personal data or enterprise workflows.
Should this block launch?
Not automatically. It should shape the launch: narrower scope, limits, monitoring, fallback behavior or a review before wider rollout.
What is the best time to run this?
Before a public launch, enterprise rollout, compliance review, funding diligence or a rewrite of the AI pipeline.
Start a conversation
Tell us what you are building, or what is currently breaking. You will get a straight answer from an engineer, not a sales script.