Customer Experience Agents in Production: The Part Nobody Budgets For
An AI agent is a system you operate, not a project you launch. Here is the loop, who owns each part of it, and what to ask before you buy.
The hard part of a CX agent is not building it. It is everything after launch: the test set it must pass before each change, the daily review of what it said to customers, and the loop that turns failures into fixes. Most CX teams budget for the build and not the loop.
“Customer experience agents in production” is a phrase LangChain, the company behind the LangGraph framework, used as the title of a 2026 guide. It is worth borrowing, because it reframes the question most CX teams are asking. The question is not “which AI agent should we buy?” It is “who runs it on Tuesday morning when it starts giving the wrong refund window, and how would we even know?” The guide’s central claim is that building the agent is no longer the hard part. The hard part is everything after launch.
That claim should sound familiar. Contact centres have run exactly this loop for decades: record the calls, score a sample against a scorecard, calibrate the scorers, coach the agent, change the script. The engineering version has new words for each step (traces, evals, judges, annotation queues) but the discipline is the one CX operations already owns. The problem is that AI agents are usually bought as a launch, with the loop left to whoever has time.
This guide translates the loop into CX terms, shows what three companies running it at scale actually measured, names the places it breaks, and ends with the questions to put to any vendor before you sign. It sits alongside the AI-native CX stack guide, which covers the architecture. This one covers the operating model.
What “In Production” Actually Means
A customer experience agent is in production when it talks to real customers with real consequences, and the team’s main job has shifted from building it to improving it. “Production” is not a status you reach at go-live. It is the state in which every change (a new prompt, a knowledge-base edit, a model update the vendor pushes on a Thursday) is a change to a live service.
The engineering view of this is a loop with two halves. In development, the agent is built and then run against an evaluation harness: a fixed set of tasks, customer personas and pass/fail scorers. If it passes, it launches. In production, every conversation is recorded, a model scores a sample of them, humans score a smaller sample to keep the model honest, and the failures flow back into the harness as new tests. The best conversations can eventually train the next model version.
The reason this lens matters now is the gap between how agents are bought and how they survive. Gartner predicted in June 2025 that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. In the same release, only 19% of 3,412 polled organisations said they had made significant investments. Most teams are early, and the ones that fail tend to fail for operating reasons, not model reasons.
The one genuinely new idea: the launch gate is a dataset, not a demo. A vendor demo shows the agent on conversations the vendor chose. A harness runs it on conversations you chose, including the angry, incoherent and borderline-fraudulent ones, and reports pass or fail per behaviour. Everything else in the loop is contact-centre QA with better tooling.
The Loop, Translated for CX Teams
The table maps each box in the engineering diagram to the thing a contact centre already does, and to who should own it. The owner column is the point. If every row says “the vendor”, you have bought a service you cannot operate.
| Stage | Engineering term | Contact-centre equivalent | Owner |
|---|---|---|---|
| Agent engineering | System prompt, context management, retrieval (RAG), tool definitions, orchestration | The brief: what the agent knows, what it may do, what it must never do | CX ops with the knowledge manager |
| Eval harness | A versioned set of tasks, datasets, personas and scorers the agent runs against | The QA scorecard plus a fixed mystery-shopper set, run before every release | QA lead |
| “Can we launch?” | Pass or fail on the harness decides whether the change ships | Go-live sign-off. A demo is not evidence; the test set is | The CX owner, not the vendor |
| Online eval | Every live conversation is traced; a model scores a sample; humans annotate a smaller sample | Call recording, QA monitoring and calibration sessions, applied to the agent | QA team |
| Continuous improvement | Failed conversations become new test cases, prompt fixes and routing changes | The coaching loop. The failure goes into the scorecard so it cannot recur silently | CX ops with engineering or the vendor |
| Model training | Fine-tuning on conversations that passed; preference tuning on human-ranked pairs | The best-call library used in onboarding, but it trains the model instead of new hires | Engineering. Most CX teams will not own this step |
Two rows are routinely skipped. The eval harness, because a demo felt like enough, and the human annotator, because the vendor’s automated score felt like enough. The automated score is only as good as the humans who calibrate it. Lyft calibrates its judges against human labels; Fastweb + Vodafone has business staff read the overnight results every day. Neither trusts the model to grade itself unsupervised.
Three Companies Running the Loop
All three examples come from LangChain’s guide and the user stories it links to, so they are vendor-published and should be read that way. They are still the most specific public numbers on operating CX agents at scale, and the lessons are consistent with each other.
Lyft
Ride-hailing, around 79 million trips a month. AI Assist for riders and drivers, built on a self-serve agent platform.
- • Roughly 270,000 customer interactions a month across seven or more production agents.
- • Reports a 65% deflection rate and a 35% AI resolution rate, and treats them as different numbers.
- • Time to build a new agent fell from about six months to about two weeks once the platform existed.
- • New agents go to 5–10% of traffic first. An on-call engineer is paged if errors or latency cross a threshold.
- • Judges return pass or fail, not a score out of ten. Generic rubrics such as helpfulness produced numbers nobody could act on.
- • The first simulated customers were too polite and passed more than 90% of the time. The simulator was retrained on real customer language with harder personas.
The lesson: Score narrow behaviours as pass or fail, not broad qualities out of ten. Report resolution and deflection separately, because a deflected customer who phones back was not resolved.
Fastweb + Vodafone (Swisscom Group)
Italian telecom. A customer-facing assistant and an internal assistant for call-centre consultants.
- • Super TOBi serves nearly 9.5 million customers in the app and on voice.
- • Reported 90% correctness, 82% resolution and a Customer Effort Score of 5.2 out of 7.
- • An internal Super Agent assists call-centre consultants; one-call resolution is above 86%.
- • Every night the day’s conversations are collected and scored 1 to 5, with an explanation and the guideline that was violated.
- • Business stakeholders, not engineers, review that output daily and agree prompt changes with the technical team.
- • Procedures written by business specialists in templates reach production within hours.
The lesson: The daily review belongs to the business. Observability was built in from the first day of development, which is why the daily scoring pipeline was possible at all.
LATAM Airlines
Largest airline in Latin America, 87 million passengers a year on 3–5% margins. Over 120 generative AI products on an internal platform.
- • The Concierge travel agent reached 52,000 users in its first beta month and now has about 4,000 daily active users.
- • Reading individual conversations showed each specialist formatting its own output, adding about 15% in latency and tokens. Formatting once at the end cut cost by about 15%.
- • 13% of messages were classed as out of scope. Review showed 95% of those were legitimate needs: check-in, baggage, the loyalty programme.
- • Adding a customer-care specialist cut out-of-scope to 1% and lifted the return rate by 6%. About 12% of daily messages now go to it.
The lesson: Out-of-scope is a demand signal, not an error rate. The architecture that works is the one production traffic reveals, not the one drawn before launch.
Where the Loop Breaks
The loop breaks in five predictable places. Four of them come straight from the companies above, who hit them and wrote about it. The fifth is the one that makes the news.
If you cannot open a conversation and see what the agent retrieved, which tool it called and why it answered as it did, you cannot run any of the loop. Fastweb + Vodafone’s chat engineering lead put it bluntly: you can’t run agentic systems in production without deep observability. Lyft traces every invocation in development, staging and production, and every production agent has an automated judge on live traffic. A vendor that offers a CSAT widget and a transcript export is offering a tenth of this.
Lyft started with helpfulness and naturalness scored out of ten and found the numbers did not tell anyone what to fix. A 6.8 for helpfulness has no owner. “Did the agent escalate when the customer mentioned a safety issue: fail” has one. Behaviour-specific pass/fail rubrics are harder to write and far easier to act on.
Lyft’s first simulated users cooperated, stayed on topic and passed the agent more than 90% of the time. Real customers interrupt, repeat themselves, give wrong details, change their mind and get angry. Until the simulator was retrained on real customer language with difficult personas, the pre-launch pass rate predicted nothing.
Lyft reports 65% deflection and 35% AI resolution for the same agents. The gap is customers who left the chat without an answer and came back by phone, email or a second chat. A vendor quoting one number is almost always quoting the first. Ask which one it is, and ask whether repeat contact within seven days is measured across channels.
In 2024 Klarna said its assistant did the work of 700 agents and handled 75% of chats. In 2025 its CEO told Bloomberg the result was lower-quality service and the company began hiring humans again. Nothing in the public account mentions a test set, a daily review or a calibrated scorer. The agent was scaled on a cost model, and the quality signal arrived from customers rather than from the loop.
The Operating Cadence
An agent in production needs a standing cadence, the same way a human team needs a QA schedule. This is the minimum. It is modelled on what Fastweb + Vodafone and Lyft describe, trimmed to what a CX team can run without an ML engineer on staff.
| When | What happens | Who |
|---|---|---|
| Daily | Read the overnight scores on a sample of conversations. Route every failed conversation to a review queue. Scan the escalation and out-of-scope lists. | QA analyst |
| Weekly | Calibration: humans score the same 20–30 conversations the model scored. Where they disagree, fix the rubric or the judge, not the humans. | QA lead with the engineer or vendor contact |
| Per change | Any prompt, knowledge, tool or model change runs the full harness first. Ship only on pass, then to 5–10% of traffic, with a rollback ready. | CX owner signs off |
| Monthly | Aggregate the month: top failure modes, resolution versus deflection, repeat-contact rate, cost per resolved conversation. Add the month’s failures to the test set. | CX ops lead reports to the business |
| Quarterly | Refresh personas and tasks from real transcripts. Retire tests the agent has passed for a year. Revisit scope using the out-of-scope list. | CX owner with product |
How to size it: give the agent its share of QA capacity in proportion to the volume it handles. If three QA analysts cover a hundred human agents and the AI agent now takes 40% of contacts, it does not need zero analysts. It needs the daily and weekly rows above to have a name against them, and the business case needs that cost in it. The agent that saved 700 headcount on paper and had nobody reading its conversations is the Klarna story.
Eight Questions to Ask a Vendor Before You Buy
Every question below maps to a stage of the loop. A vendor that answers all eight well is selling you something you can operate. A vendor that answers the first one badly is selling you something you can only watch.
- 01
Can we open any conversation and see every step: what was retrieved, which tool was called, what the model saw?
Without traces there is no loop. Everything below depends on this.
- 02
Can we bring our own test conversations and run them before every change, including the model updates you push?
The launch gate must be ours. A vendor-chosen demo set tests what the vendor is good at.
- 03
How are live conversations scored, on what sample, and who calibrates the scorer against human judgement?
An uncalibrated automated score drifts with the model it is grading.
- 04
Can our QA team label conversations inside the tool, and do those labels become test cases automatically?
If labels sit in a spreadsheet, the loop has a manual step that will be skipped under pressure.
- 05
What is your definition of “resolved”, and can we measure repeat contact within seven days across channels?
Deflection and resolution are different numbers. You are paying for the second.
- 06
What happens when the underlying model changes? How much notice do we get, and do our tests re-run?
Model updates are production changes to your service, made by someone else.
- 07
Can we roll a change out to a slice of traffic and roll it back without a ticket?
Staged rollout is how Lyft ships. A big-bang deploy means a big-bang failure.
- 08
If we leave, what do we take with us: the conversations, the test set, the labels, the rubrics?
The loop’s assets are the thing you built. They should not be locked to one vendor.
The honest summary: “in production” is a verb. It describes work a team does every day, not a milestone a project reaches. The CX teams that get value from agents are the ones that budgeted for the loop, staffed it, and kept the launch gate in their own hands. The ones that did not are the ones Gartner is counting.
Related Reading
Sources
- LangChain, Customer Experience (CX) Agents in Production: Lessons from Lyft, Vodafone, and LATAM Airlines (2026)
- LangChain, Lyft built a self-serve AI agent platform for customer support
- LangChain, Fastweb + Vodafone user story
- Gartner, press release, 25 June 2025: over 40% of agentic AI projects will be canceled by end of 2027
- Klarna figures: Bloomberg interview with CEO Sebastian Siemiatkowski, May 2025, as reported by Entrepreneur and Silicon Canals.
AI Agent, Support and Knowledge Tools
Chatbot and agent platforms, support desks, automation and knowledge tools from the directory. Put the eight questions above to each of them:
Ada
AI chatbot platform for customer support, with no-code builder and focus on deflecting common questions.
Buyer fit
Enterprise, Mid-Market teams that need AI-Driven Training and Suggestions and Customer Segmentation coverage.
Key Features
- AI-Driven Training and Suggestions
- Customer Segmentation
- Intent Recognition
- NLP Engine
Google Dialogflow
Google conversational AI platform for voice and chatbots with developer-friendly interface and NLU.
Buyer fit
Enterprise, Mid-Market, Small Business, Startup teams that need Intent Recognition and NLP Engine coverage.
Key Features
- Intent Recognition
- NLP Engine
- Sentiment Analysis
- Bot-to-Human Handoff Orchestration
IBM Watson Assistant
Enterprise conversational AI with NLU for intent detection, entity extraction, and backend system integration.
Buyer fit
Enterprise, Mid-Market teams that need Intent Recognition and NLP Engine coverage.
Key Features
- Intent Recognition
- NLP Engine
- Sentiment Analysis
- Bot-to-Human Handoff Orchestration
Kore.ai XO Platform
Suite to design, test, and deploy AI assistants across channels with natural language understanding.
Buyer fit
Enterprise teams that need Intent Recognition and NLP Engine coverage.
Key Features
- Intent Recognition
- NLP Engine
- Sentiment Analysis
- Bot-to-Human Handoff Orchestration
Microsoft Power Virtual Agents
No-code/low-code chatbot platform integrated with Microsoft Power Platform and Azure AI.
Buyer fit
Enterprise, Mid-Market teams that need Intent Recognition and NLP Engine coverage.
Key Features
- Intent Recognition
- NLP Engine
- Bot-to-Human Handoff Orchestration
- Human Escalation
Rasa
Open-source conversational AI framework for developers wanting full ML control and data privacy.
Buyer fit
Enterprise, Mid-Market, Startup teams that need Intent Recognition and NLP Engine coverage.
Key Features
- Intent Recognition
- NLP Engine
- Bot-to-Human Handoff Orchestration
- Multilingual Support
Frequently Asked Questions
What does “customer experience agents in production” mean?
A customer experience agent is in production when it handles real customer conversations with real consequences, and the team’s main work has shifted from building it to improving it. The phrase comes from a 2026 LangChain guide whose central claim is that building the agent is no longer the hard part; operating it is. Operating means a test set the agent must pass before every change, recorded conversations scored by a model and checked by humans, and a loop that turns failures into new tests and fixes.
What is an eval harness for a CX agent?
An eval harness is a versioned set of test conversations (tasks), customer personas, datasets and pass/fail scorers that an AI agent runs against before any change ships. It is the equivalent of a contact centre QA scorecard combined with a fixed mystery-shopper set. Lyft describes its harness as tasks, datasets, personas and scorers, and uses the result as the launch gate: if the agent fails, the change does not go to customers.
What is the difference between deflection and resolution for an AI agent?
Deflection counts conversations the agent handled without a human. Resolution counts conversations where the customer’s problem was actually solved and they did not come back through another channel. Lyft reports 65% deflection and 35% AI resolution for the same agents. The gap is customers who left the chat and phoned, emailed or returned. Vendors quoting a single number are usually quoting deflection; the business case should be built on resolution.
What is LLM-as-a-judge in customer service?
LLM-as-a-judge means using a language model to score live or test conversations against a rubric, so that every conversation can be evaluated instead of a tiny sample. It only works when humans calibrate the judge: scoring the same conversations and fixing the rubric where they disagree. Lyft found generic rubrics such as helpfulness out of ten produced unactionable numbers and moved to behaviour-specific pass/fail checks. Fastweb + Vodafone scores every day’s conversations overnight and has business staff review the results each morning.
How much QA does an AI customer service agent need?
An AI agent needs QA capacity in proportion to the volume it handles, not zero. The minimum cadence is a daily read of automated scores with failed conversations routed to review, a weekly calibration session where humans score the same conversations the model scored, a full test run before every prompt, knowledge, tool or model change with a staged rollout, a monthly aggregate of failure modes and resolution versus deflection, and a quarterly refresh of personas and scope.
Who should own an AI agent after launch?
The CX organisation should own the launch gate, the test set, the daily review and the definition of resolved, with engineering or the vendor owning the build, the tracing infrastructure and any model training. In the companies with published results, business stakeholders read the daily evaluation output and agree changes with the technical team. If every operating step is owned by the vendor, the buyer has a service they can watch but not run.
The Companion to Part 6
Part 6 of the CX Field Guide covers what an AI-native CX stack looks like and how to get there. This guide covers what it takes to keep one running once it is live: the test set, the daily review, the calibration, and the questions that separate a vendor you can operate from one you can only watch.
View All Insights →