How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Zapier is the #1 workflow automation platform for small and midsize businesses, connecting to more than 7,000 of the most popular work apps. We were also one of the first companies to build and ship AI features into our core products. We've had the opportunity to work with Braintrust since the early days of the product, which now powers the evaluation and observability infrastructure across our AI features. We’ll walk through a couple of our projects – AI Zap Builder and Zapier Copilot – and how we built them from the ground up with evals and observability in mind. Hopefully, you can walk away with a few of our learnings from operating these projects at scale while systematically improving their performance. Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025 About Ankur Ankur Goyal is the founder & CEO of Braintrust—the developer platform that companies like Zapier, Notion, Instacart, Airtable, and more use to evaluate, log, and ship reliable AI products to millions. He was previously Head of AI platform at Figma, founder and CEO of Impira, and VP Eng at Singlestore. After Figma acquired Impira, he led the AI team there, and saw a number of the same blockers to AI development at Impira, Figma, and other peer companies, which led to founding Braintrust About Olmo Olmo Maldonado is a Senior AI Engineer at Zapier, where he develops high-scale AI services aimed at democratizing automation. He leads the team behind the AI Zap Builder and Copilot, and has created shared services that have enabled multiple teams to scale LLM usage to production loads. Under his leadership, Zapier improved AI accuracy by fostering a culture of Eval Driven Development. Olmo is a former Googler and a core developer of MooTools. He holds a Master’s degree in Electrical Engineering from the University of California,…