Comcast ran 140 million simulated calls before shipping its CX agent

Comcast built a simulation system to test its customer-experience AI agent before changes go live. The system replays real past conversations against a new version of the agent. A team of judges, some human and some automated, scores the results. If the new version performs worse, the team blocks the change before any customer sees it. The paper reports results at 140 million interactions. At that scale, a single bad prompt change could harm millions of customers. Simulation gave the team a way to measure harm before it happened. A/B results confirmed that changes passing simulation performed better in production. Changes that skipped simulation did not perform as well. The core design tension is speed versus safety. Teams want to ship fast. Customers need the agent to behave well. Simulation lets a team move with confidence before they approve a release. A release decision on a production agent needs the same evidence base as any other quality decision. That evidence has to exist before the customer ever sees the change.