Your agent works in the demo. Does it work in production?
We test AI agents the way attackers and real users will, score them against an open rubric, and fix what we find: eval suites that gate CI, guardrails, tracing and cost controls. You decide how much of it goes into your code, starting from none.
The pilot wall
Why agents stall before production
Most teams we meet have built a first agent. Few can answer these questions with evidence.
When the agent gets it wrong, can you tell?
Without an eval suite, bad answers show up as customer complaints instead of CI failures.
When a prompt or model changes, what breaks?
Prompts get edited in production and providers retire models on their own schedule. Nobody knows what regressed until a user notices.
Where did this month's token bill go?
Spend jumps and nobody can point to the feature, tenant or model call responsible.
When someone attacks it, what happens?
Prompt injection hidden in a web page, a document or a tool result. Without input checks and tool limits, the damage is whatever the agent is allowed to do.
When it does something odd, can you see why?
Without traces of each model call, tool call and guardrail decision, debugging means rerunning the conversation and hoping it happens again.
Can you show a reviewer it's ready?
Risk, security or compliance asks for evidence before go-live, and the team has a demo and a good feeling.
What we put in place
Six pieces, sized to your agent
They work with the framework you already use, such as LangGraph, Google ADK, Pydantic AI or the OpenAI Agents SDK, or with an agent in any language behind an HTTP endpoint.
Golden cases for known-good behaviour and adversarial cases for attacks. Every change to a prompt, tool or model runs them, and a regression blocks the merge.
Single-turn and multi-turn attacks grouped into six classes from Google DeepMind's AI Agent Traps research. Each class has its own pass rate and threshold, so one strong score can't hide a weak one.
PII redaction and injection scanning on input, provenance checks on retrieval, policies on tool calls, generated SQL, code and shell commands, and human approval where the stakes call for it.
Model calls, tool calls and guardrail decisions as OpenTelemetry gen_ai spans, in the backend you already run: Langfuse, Logfire, Datadog, Application Insights or CloudWatch.
Token and dollar cost per request, by tenant, feature and model, with soft or hard budgets. Cheaper-model routing where your evals show quality holds.
Production traces reviewed against what the agent was told to do. Recurring failures become regression cases, and model upgrades are replayed on real traffic before they ship.
Your code, your call
Nothing has to go into your codebase
Outside-in
We test your staging endpoint over HTTP, in any language, and read a trace export. Nothing is installed on your side.
In your pipeline, not your product
Test suites live in a separate repository or run as a container in CI. Your production image doesn't change.
A guardrail gateway
Point your model SDK at a proxy that enforces guardrails. One base URL changes; no library runs in your process.
In your code, when you want it
For the deepest control, guardrails go into your framework's hooks. Everything at this level is removable.
The rubric
Score your own agent against the open scorecard
Eval coverage (20%)
Golden and adversarial datasets, per-step evals, calibrated LLM judges.
Regression detection (15%)
Pinned models, and evals that block any prompt, tool or model change that regresses.
Guardrails (15%)
Input, output and tool-call checks, with human approval on high-stakes actions.
Observability (15%)
OpenTelemetry gen_ai spans, linked to users and sessions.
Cost controls (10%)
Per-request budgets, attributed to tenant, feature and model.
Failure-mode mapping (10%)
A written list of failure modes, each with detection and a mitigation.
Security and data handling (10%)
Redacted PII and secrets, least-privilege tools, all six attack classes tested.
Deployment and operations (5%)
Versioned config, staged rollouts, an incident runbook and a quick rollback.
Ways to work with us
Reliability Snapshot: 3 to 5 days, fixed price
An outside-in test of one agent through its staging endpoint. You get the scorecard dimensions we can evidence, pass rates for each attack class, your top five risks and a readout with a live demo. No access to your code is needed.
Reliability Audit: 2 weeks, fixed price
The full scorecard with evidence from your code, traces and evals, an adversarial test set built for your agent, and a prioritised 30/60/90-day roadmap.
Reliability Hardening: 6 to 10 weeks, fixed price
We implement the roadmap: eval suites and CI gates, guardrails, tracing and cost controls, at the level of involvement you choose. The closing readout replays the audit's attacks to show what is now stopped.
Reliability Retainer: monthly
Production traces turned into regression cases, new attack techniques added as they are published, and a monthly scorecard showing what changed.
Model Migration Check: 1 week, fixed price
Your provider is retiring a model, or a cheaper one looks tempting. We replay real traffic on the new model and give you a go or no-go with per-case differences, cost and latency.
Open source
The scorecard and eval harness are open source
The Agent Reliability Kit is the rubric and eval runner we use on engagements, under the Apache-2.0 licence. Run it yourself, gate your own CI with it, and keep it whether or not you work with us.
Run JSONL test suites against a Python function or any HTTP endpoint and fail the build when pass rates drop.
Tag adversarial cases by trap class and set a pass-rate threshold for each one.
Fill in the eight dimensions with evidence and render a self-contained HTML report you can diff over time.
FAQs
Common questions
Do we have to add your code to our codebase?
No. A Snapshot tests your agent over HTTP and reads exported traces, with nothing installed on your side. Guardrails can run in a gateway in front of your model provider. Going deeper is your choice, and everything we add can be removed.
How is this different from Langfuse, Braintrust or LangSmith?
Those are good tools and we work with them: results can go to any of the three. What teams usually lack is the engineering around them: test sets that target their failure modes, gates in CI, guardrails, and someone reviewing production traces every month.
Do you replace our agent framework?
No. We work around the framework you chose. We have working integrations for LangGraph, Google ADK, Pydantic AI and the OpenAI Agents SDK, recipes for others such as Agno, and anything else can be tested over HTTP.
Who owns what at the end?
Everything built for you is yours: test datasets, scorers, guardrail policies, configuration and reports. Our tooling stays ours, and you get a perpetual, royalty-free licence to keep using and modifying what we deliver, inside your organisation, after the engagement ends. The scorecard and eval harness are open source under Apache-2.0.
Does this help with the EU AI Act or OWASP?
It produces evidence a compliance team can use: we map our controls to the OWASP Top 10 for LLM and agentic applications, the NIST AI RMF, ISO/IEC 42001 and the EU AI Act's requirements on risk management, logging, human oversight and robustness. It doesn't make a system compliant on its own, and it isn't legal advice.
Is the Snapshot a sales call?
No. It is a paid, fixed-price piece of work with a written result: the scorecard dimensions we could evidence, attack-class pass rates, the test set we ran and your top risks. You keep all of it.
Will you sign an NDA before seeing our system?
Yes, before any access. Traces containing personal data are handled under a separate data-processing addendum.
Our agent runs on AWS, Azure or Google Cloud. Does that matter?
No. We read traces from CloudWatch, Application Insights and Logfire, and from any OpenTelemetry backend, and we have delivered agents on all three clouds.
Start with a Snapshot
In 3 to 5 days, with nothing installed on your side, you find out how your agent holds up against real attacks and where it stands on the scorecard. The results are yours whether or not we do the fixing.