Your Agent Fix Just Broke Something Else – Sofia Jakovcevic, Arize AI
Episode details
Episode description
A one-line prompt change made a support agent 17 points safer and 17 points worse at finishing tasks. Sofia Jakovcevic, forward deployed engineer at Arize AI, explains why agents fail at runtime rather than in code, and walks through a five-step regression loop built from production traces: curate datasets, score with evaluators, run baseline experiments, compare changes, and gate merges in CI. Then she shows how to close the loop at agent speed, with coding agents that read traces and fix themselves. Chapters 00:00 Intro: observability for agents 01:08 The real problem: fixes that break other things 02:26 A one-line prompt change: +17 and -17 04:48 Agents fail at runtime, not in code 06:25 The five-step regression loop 06:48 Step 1: Build datasets from production traces 07:50 Step 2: Score with evaluators 09:48 Steps 3-4: Baseline experiments and comparisons 11:14 Step 5: Gate merges in CI 11:48 Demo: experiments in Arise 15:38 Using coding agents and skills 16:12 Closing the loop at agent speed Sofia Jakovcevic, Arize AI Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Watch also from the same channel
More episodes from this channel
You might be interested in
