Most blogs about building AI products are written by the engineers who built them. Our tech lead wrote excellent one about how we rescued a regressing voice agent - eval harnesses, model drift, latency traces, the whole story. It's worth your time, so take a look.
This is the same project from the project manager's seat - and it has a moment in it that changed how I think about shipping software.
The launch that went well
The product was a voice agent that answers a restaurant's phone, takes pickup orders with modifications, books reservations, and handles menu questions. We took it from whiteboard to a production SaaS platform in a few months with a 3-4 person team - an iPad app, a web admin, Stripe billing, EU phone provisioning, the agent itself and a simple landing page.
The delivery went the way you'd want. We ran structured discovery with the client to map the real calls the agent would face, designed and prototyped before writing code, and kept a shared Slack channel where the client tested each iteration against real restaurant scenarios. That last part mattered more than any spec. Voice agent quality can't be read off a document - it has to be heard, and the client hears a hundred real food orders a day in a way we never could. Their ear caught things our testing never would.
After a few iterations, we shipped. The client called the agent "fantastic." From a project standpoint, this was a clean win.
When nothing changed and everything broke
A couple of weeks after go-live, the agent started getting orders wrong. Not subtly - it would take "two sandwiches and a salad" and log eight of something else completely. That would mean customers on the line, orders mangled, a client who'd been delighted now watching his phone agent possibly embarrass him. After a mild panic, we refocused on the issue.
The first thing we did was the obvious thing: check what we'd changed. The answer was nothing. Our code hadn't been touched in three weeks. No deploy, no edit, nothing on our side.
In eight years of shipping software, "we changed nothing and it broke" had always meant someone was mistaken. This time it was just true. BUT we quickly realized, the agent runs on managed AI services from outside vendors, and the model powering it can be updated without us being told. What we'd tested and signed off on wasn't what was running anymore. The phenomenon has a name - model drift - and for a PM it's a new kind of risk: something you accepted as finished can quietly stop working, and none of the usual signals will point you to it.
The scramble that didn't scale
So we started fixing it the way we'd tested the agent previously: make a change to how it was instructed, call it, listen, decide whether it sounded better, change something else, call again.
Testing by ear works for the first few rounds. It completely falls apart the moment you're trying to hold quality steady across a dozen different call types, each with its own ordering logic and edge cases. Every fix risked silently breaking something we'd already gotten right, and we had no way to see it happen. We were making changes, we were busy, and we could not tell if we were getting better or just moving the problem around.
For a PM, that's the worst place to be. You're spending time, the team's spending time, and there's no signal telling you whether any of it is working.
The turn
The fix that actually worked wasn't a cleverer prompt. It was building a way to measure. Yes, that’s 101 of any development, but at this point in this field it was still quite new for us.
The team built an automated test harness from the client's real phone scenarios and gave every change a score instead of a gut feeling. (Jan's post covers the engineering - that's the part I pointed you to him for.) The effect on the project was immediate. "Does this feel better?" became a number. A change that fixed one thing and broke two others showed up right away instead of surfacing weeks later as a client complaint. We went from intuitive tinkering to actually having a process, after new iterations we quickly realized that this was the missing piece.
We got the agent back to full reliability - and this time we could see the numbers, not just say "trust us, it's better."
That harness did something beyond fixing the immediate fire. It turned the drift problem from a recurring nightmare into a managed risk. A scheduled run of those same scenarios against the live agent works as an early-warning system - the one thing you actually control when the model itself is out of your hands.
What I carry into every AI project now
The drift incident deepened my understanding of AI products and also reminded me of the basics of the whole job. With traditional software, go-live is roughly the finish line. With an AI product built on managed services, it isn't - the product can regress on its own, so monitoring belongs in the plan from day one, not bolted on after the first surprise. We now raise it during scoping, and being able to detect and correct drift quickly is part of what shipping AI responsibly means.
And underneath that sits the thing the project reminded me: with a probabilistic product, investing early in a way to measure quality isn't a technical nice-to-have. It's the thing that makes the work manageable at all.
