Your AI feature works in the demo. Can you prove it works in production?
A 10-business-day review of the engineering around an AI feature: what it retrieves, what it logs, what it costs, what happens when the model is wrong, and whether permissions survive contact with real users.
Talk to an engineerWhen this review is useful
- The prototype answers well in demos, but no one knows how often it is wrong.
- Prompt changes ship without a regression suite, so quality moves by anecdote.
- The system answers from private documents, but retrieval permissions are unclear or untested.
- The bill arrives before anyone knows the cost per request, per customer or per feature.
- Answers are stored, but not the sources, model, prompt or index version that produced them.
- An agent can call tools, but retries, limits and attribution were not designed as part of the workflow.
What we review
Retrieval quality
Chunking, hybrid search, ranking, reranking, duplicate context and whether each answer can point back to evidence a user can inspect.
Evaluation
A fixed question set, expected properties, regression checks before deployment, and cost/latency tracked with each change.
Permissions
Whether retrieval filters by the requesting user before the model sees context, and whether tests prove that boundary.
Observability
Prompt, model and retrieval index versions recorded with outputs, plus traces across app, retrieval, model call and tools.
Cost and latency
Token budgets, model routing opportunities, caching, worst-case request behavior and where spend becomes visible.
Failure behavior
Timeouts, unavailable models, missing evidence, unsafe tool arguments, malicious input and what the product does instead of pretending nothing happened.
How the review runs
Days 1-2: scope the critical path
Pick the AI workflow where being wrong is most expensive: customer-facing answers, internal decision support, document processing or agent actions.
Days 3-6: inspect and test
Read the implementation, sample real traces where possible, exercise permission and failure edges, and identify what cannot currently be proven.
Days 7-9: write the report
Prioritise findings by business impact and engineering effort, separating quick wins from structural work.
Day 10: handover
Walk through the findings with engineering and product so the report turns into decisions, not shelfware.
What you get
- Written report with prioritised findings across quality, privacy, cost, latency and operability
- Evidence gaps: what the system can prove today and what it cannot
- Risk matrix for retrieval, permissions, evaluation, observability and agent actions
- 30-day action plan with quick wins separated from structural work
- Handover call with the people who will act on the findings
Questions we get asked
Is this prompt tuning?
No. Prompt tuning may be one recommendation, but the review is about production readiness: retrieval, evaluation, permissions, cost, latency, observability and failure paths.
Do you need production write access?
No. The review is designed around read-only access, source code, architecture context, dashboards and conversations with the engineers who know the system.
What if the feature is still a prototype?
That is often the cheapest moment to review it. If records, permissions and evaluation are designed before launch, they are decisions. Added later, they become migrations.
Can you implement the fixes afterwards?
Yes, if there is a fit and availability. The report is still yours either way: your team, another vendor or Devsitia can execute it.
Start a conversation
Tell us what you are building, or what is currently breaking. You will get a straight answer from an engineer, not a sales script.