Agentic AI in Production: The Questions B2B Buyers Should Ask Before They Sign

"Powered by AI agents" is on almost every B2B SaaS homepage now, the way "blockchain-enabled" was in 2018 and "AI-powered" was in 2023. Some of it is real production infrastructure that saves customers meaningful time and money. A lot of it is a chat window in front of a single API call, marketed as autonomy. As a team that builds both the agentic features and the systems they plug into, here is the checklist we would actually use if we were the ones buying.

1. What happens when the agent is wrong?

Ask for a specific, recent example of the agent failing in production — not a hypothetical. If the vendor cannot produce one, either the product has not seen real production traffic, or they are not tracking failures closely enough to know. Then ask what the blast radius was: did a human catch it before a customer did? Was there a rollback path, or did someone have to manually undo damage?

2. Is there a human-in-the-loop option, and is it actually enforced?

Many vendors will say "yes, you can review before it executes" — but check whether that review step is the default or an opt-in buried in settings, and whether it is enforced for the specific actions that matter to your business (sending money, sending customer-facing emails, modifying records with legal/compliance weight). A review checkbox that is off by default is not a safety feature; it is a legal disclaimer.

3. What is the actual data boundary?

Which of your data does the agent send to a third-party model provider, and is that documented in the DPA, not just in a blog post? For EU buyers this is not optional — ask for the sub-processor list, ask where inference happens (region matters for GDPR), and ask whether your data is used for any kind of model improvement/training by default versus opt-in.

4. How does cost behave under load or misuse?

Agent loops that retry, re-plan, or chain multiple model calls per user action can have highly variable cost per request compared to a traditional API call. Ask for real cost-per-successful-outcome numbers, not cost-per-API-call — and ask what happens (rate limiting? hard cap? silent cost overrun?) if a user or a bug triggers a runaway loop.

5. What is the actual autonomy level, in plain language?

"Autonomous" gets used for everything from "fully unsupervised, multi-step, tool-using agent" down to "we call an LLM once per form submission." Ask the vendor to describe, step by step, what happens between a user action and a completed task, and count how many of those steps are deterministic code versus model-generated decisions. That number tells you more than any marketing claim.

6. What breaks first if the underlying model changes?

Model providers deprecate and change model versions on their own schedule, not yours. Ask how the vendor handles a model deprecation or a behavior-changing update — do they pin versions, run eval suites before rolling forward, and can they roll back if a new model version regresses on your specific use case?

Why we ask ourselves the same questions

We build agentic features into client products, and we hold our own work to this checklist before it ships. If a client asks us "what happens when it's wrong," we want a real answer with a real example — not a reassurance. That is the bar we think B2B buyers should hold every vendor to in 2026, including us.