Know if the change made it better, before your users do
Every AI feature eventually faces the same question: is this new prompt actually better, or does it just feel better? Without a test suite, the answer is somebody's opinion, and the regression ships.
Talk to an engineerHow you know you need this
- Someone improved the prompt last week and nobody can prove it helped.
- A model version changed under you and the quality moved, but you found out from a customer.
- Testing a change means a person trying five questions by hand and forming an impression.
- The team is afraid to touch a prompt that works, so the feature has quietly stopped improving.
- Cost per request grows and nobody can attribute the increase to a specific change.
What it gives you
A fixed question set that reflects reality
Built from your real traffic and your real edge cases, including the ones that already embarrassed you, with the answers you expect.
A score on every change
Prompt, model, retrieval settings, chunking: change any of them and see the effect on correctness, grounding and refusal behaviour before it ships.
Cost and latency alongside quality
Because an answer that is two percent better and four times more expensive is not an improvement, and that trade-off should be visible when the decision is made.
Runs in CI, like any other test
A pull request that degrades the AI feature fails, the same way one that breaks a unit test does. That is what turns discipline into infrastructure.
Traces you can actually read
When the score drops, you need to see what was retrieved and what the model did with it. A number without an explanation just tells you to panic.
Who it is for
Teams with an AI feature already in production
It works, it matters, and every change is currently a leap of faith.
Teams about to change model or provider
Model migration without an evaluation set is a rewrite with no tests. The suite pays for itself on the first migration.
Teams whose AI costs are growing
Cost per request tracked per change turns a quarterly surprise into an engineering decision.
Teams that had a public failure
Something went wrong in front of customers, and the fix has to be verifiable rather than promised.
What you get
- An evaluation suite built from your questions and your edge cases
- Scoring for correctness, grounding, refusal behaviour, latency and cost
- CI integration, so a regression blocks a merge instead of reaching users
- A baseline measurement of where you are today, which is often the most uncomfortable and most useful deliverable
- A short guide for your team on adding cases as new failures appear
Questions we get asked
Can you evaluate a non-deterministic system at all?
Yes, statistically rather than exactly. You do not assert that one output equals one string. You score a set of cases on dimensions that matter and watch the aggregate across changes. The same reasoning applies to any system where the individual outcome is uncertain but the distribution is stable enough to measure.
How many test cases do we need?
Fewer than teams expect. Thirty to fifty well-chosen cases covering your real distribution and your known failures catch most regressions. A thousand generated cases that all look alike catch fewer, and cost more to run.
Does this work with our stack?
The harness is built around your API boundary rather than around a specific framework, so it works with whatever you call, LangChain, LangGraph, direct provider APIs or your own layer. It has to, because the point is to survive you changing that layer.
Is this worth it if the feature is small?
Proportionally. A small feature needs a handful of cases and a CI step, not a programme of work. The cost of not having it shows up the first time you change a model and cannot tell what moved.
Start with a baseline
The first useful step is measuring where you are today. It usually takes a couple of weeks, and it is uncomfortable in a productive way: most teams discover their feature is worse than they thought on cases they never tried. Tell us what your AI feature does and we will scope it.