About this channel
AI Engineer focuses on the practical difficulties of building software with AI and making agents work reliably. The channel's talks examine browser automation, model selection, memory, tool access…
AI Engineer focuses on the practical difficulties of building software with AI and making agents work reliably. The channel's talks examine browser automation, model selection, memory, tool access and the effects of generated code on development teams. Its central subjects are the gaps between an impressive capability and a system that performs consistently within a real workflow.
Developers and engineering leaders will recognise many of the questions being raised. Does a coding agent actually help a team deliver faster? Why can a browser agent with a high success rate still fail as part of a larger task? How should a system choose a model, retrieve current information or retain useful memory? The material is suited to people who need to make architecture and process decisions around those questions, with individual speakers presenting a particular argument or set of observations.
Derek Meegan's browser-agent talk tackles reliability, while Jakub Hojsan's session examines coding agents working with outdated information. A talk by Dylan Couzon turns to ownership of agent memory. These specific engineering concerns sit beside discussions of MCP, communication between agents and the limits of treating a protocol as a complete solution. Named speakers and organisations give the episodes the form of technical presentations rather than general news summaries.
The organisational strand considers evidence about AI in software development, the changing role of code review and what a chief AI officer is expected to do. That widens the channel's scope from how to construct an agent to how a team should evaluate and govern its use. Viewers can move between implementation questions and debates about productivity, using the talks to examine both the systems they build and the assumptions behind their development process.
Latest episode
Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab
Your agent scores great on benchmarks. Then real customers use it and it breaks.
Felipe Blanes from the Amazon AGI Lab shares what he learned working directly with customers of Nova Act, Amazon's…
Your agent scores great on benchmarks. Then real customers use it and it breaks.
Felipe Blanes from the Amazon AGI Lab shares what he learned working directly with customers of Nova Act, Amazon's service for building browser agents, from research preview to general availability on AWS. He calls the core problem the benchmark illusion: static evals look great until customers do things nobody expected, and then all you have left is hope. His fix is an eval flywheel built around the customer: define success the way the customer does, capture signals (including by talking to customers), diagnose gaps in the model, harness or product, and feed them into decisions. He explains the trust cliff (80% reliability feels like more work; around 92% feels trustworthy), why being open about limits builds more trust than higher benchmarks, surprising use cases from Amazon Leo, Hertz and Sola, and four principles for keeping evals fresh.
In this talk:
• The benchmark illusion, and why static evals leave you hoping
• A four-step eval flywheel driven by real customer signals
• The trust cliff: why 80% reliability isn't enough
• Why being open about limits builds more trust than higher scores
SPEAKER
Felipe Blanes, Amazon AGI Lab
LinkedIn: https://www.linkedin.com/in/felipeblanes/
LINKS
Amazon Nova Act: https://nova.amazon.com/act
Nova Act docs: https://docs.aws.amazon.com/nova-act/latest/userguide/what-is-nova-act.html
CHAPTERS
0:00 Intro
0:47 What Nova Act is
1:57 Working directly with customers
2:22 The benchmark illusion
3:21 Static evals and hope
4:26 Close the customer loop
4:41 The eval flywheel
4:56 Step 1: define success
5:11 Step 2: capture signals
5:31 Talk to your customers
5:51 Step 3: diagnose gaps
6:56 Step 4: feed decisions
7:26 Four stages of the customer journey
9:35 Lesson 1: the trust cliff
11:10 Lesson 2: be open about limits
12:45 Lesson 3: surprising use cases (Amazon Leo, Hertz, Sola)
14:44 Four principles
15:59 Evals should get smarter every week
16:14 Two takeaways
Recorded at the AI Engineer World's Fair 2026 in San Francisco.
Subscribe for more talks from the engineers building with AI.
AI Engineer: https://ai.engineer
YouTube: https://www.youtube.com/@aiDotEngineer
X: https://x.com/aiDotEngineer
LinkedIn: https://www.linkedin.com/company/aidotengineer/
#LLMEvals #AIAgents #AIEngineer
YouTube Shorts
AI Architects @ AI Engineer
AIE World's Fair 2026 Complete Playlist
Anthropic @ AI Engineer
Coding Agents @ AI Engineer
Data (Datasets and RL Environments) @ AI Engineer
FDE (Forward Deployed Engineering) @ AI Engineer
Post-training @ AI Engineer
uploads
More episodes from this channel
How a Remote Company Builds AI Fluency — Em Shreve, Automattic
8.10.2026
Handing people AI tools doesn't change how they work. Giving them real time to learn does. Em Shreve, head of AI enablement at Automattic, the company behind WordPress.com and Tumblr with over 1,400 people...
Brains vs Hands: How to Run AI Agents Safely in Production — Viren Baraiya
8.10.2026
Let the model decide what should happen next. Never let it improvise how a production cluster gets restarted. Viren Baraiya, co-founder and CTO of Orkes and the original creator of Netflix Conductor, explains how...
We Mapped 115 Microservices for Our Coding Agents — Kamalakannan Nandagopal, Postman
8.10.2026
Coding agents are great inside one repo. Postman has 115 microservices and thousands of endpoints. Kamalakannan Nandagopal, staff engineer at Postman, shares how Postman made coding agents work across a real distributed system. Docs...
Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j
8.10.2026
Agents finish a task, then forget what they learned. Retrieval alone won't fix that. Will Lyon, senior product manager at Neo4j, explains why agent memory built on embeddings and retrieval isn't enough: recall isn't...
Your AI Agent Has No Nervous System — Matt Gibiec, Dynatrace
8.10.2026
Your coding agent is fast. But it can't see how your services actually run, so it guesses. Matt Gibiec, who leads AI-native solutions engineering at Dynatrace, explains why AI agents need something like a...
Why We Deleted Our MCP Server and Rebuilt It — Abhi Arya, Reducto
8.10.2026
Reducto gave its whole team an MCP server with a tool for every endpoint. It broke the same day. Abhi Arya, who works on product at Reducto, argues that building agent-first software comes down...
Let Anyone at Your Company Ship Internal Apps with AI — Garrett Galow, WorkOS
8.10.2026
Building an app with AI got cheap. Getting it in front of your coworkers didn't. Garrett Galow from the WorkOS product team shows how WorkOS lets anyone at the company build and ship internal...
Giving AI Agents Memory That Learns — Jake Broekhuizen, LangChain
8.10.2026
Most teams are great at watching what their agents did. Far fewer turn that into lessons for the next run. Jake Broekhuizen from LangChain's Labs team explains how to give agents memory that actually...
Why Bigger Context Windows Won't Save Your Agent — Elizabeth Fuentes Leone, AWS
8.10.2026
A bigger context window won't save your agent. Past a point, it forgets the middle. Elizabeth Fuentes Leone, developer advocate at AWS, explains why agents break when tools return huge amounts of data, and...
AI Hackers Are Faster Than Your Pen Test — Eli Cohen, Snyk
7.10.2026
Attackers use AI too, and the fastest AI attack he cites took four minutes. A pen test once or twice a year can't keep up. Eli Cohen of Snyk, co-founder of the runtime security...
The Software Factory: From Bug Report to Production Code — Davis Palmie, Factory
7.10.2026
Engineers are going from typing code to governing agents. The next step is a system that turns bug reports into production code. Davis Palmie, member of technical staff at Factory, explains the software factory:...
Why Your AI Agents Can't Talk to Each Other (Yet) — Vlad Luzin, BAND
7.10.2026
If you run Claude and Codex side by side, you're the router passing messages between them. It doesn't have to be that way. Vlad Luzin, co-founder and CTO of BAND, argues the future belongs...
Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind
7.10.2026
Your agent shouldn't run on your laptop. Give it its own computer in the cloud, with one API call. Philipp Schmid from Google DeepMind walks through the Gemini Interactions API, Google's new API for...
Redesigning How Software Gets Built With AI Agents — Sonar & McKinsey Panel
7.10.2026
Most companies say they've deployed AI agents. Fewer than a third say it's paying off. What do the rest do differently? A panel with Sonar CEO Tariq Shaukat and Ali-Reza Adl-Tabatabai, former CEO of...
From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer
7.10.2026
When one agent finally cracks a hard problem, what if every agent in the company learned it too? Rafal Wilinski, founding engineer at Runlayer, shows how to go from one self-improving agent to a...
Grep or Embeddings? Agentic Search Over Company Documents — George He, LlamaIndex
7.10.2026
Grep works great on a codebase. Try it on millions of PDFs, slides and scanned schematics. George He, head of engineering at LlamaIndex, explains when agents need pre-indexed search and when plain file traversal...
Every AI Company Is Accidentally Building a Bank — Dor Sasson, Stigg
7.10.2026
In one month, three of the biggest AI companies had a pricing emergency. Dor Sasson says it was really an infrastructure problem. Dor Sasson, co-founder and CEO of Stigg, argues that every AI company...
The 6 Pillars of an Agentic Harness for Production — Varun Krovvidi, Resolve AI
6.10.2026
AI changed how we write code. But 70% of engineering time goes to running and fixing it. Varun Krovvidi, product marketing lead at Resolve AI, explains why AI that writes code well still struggles...
Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell — Akamai
6.10.2026
A small model guesses the next few tokens. The big model checks them in one pass. When is that worth it? Sheilah Kirui, developer advocate at Akamai, explains speculative decoding and how to tell...
Move Fast and Don't Break Things: Scaling Databases for the AI Era — PlanetScale
6.10.2026
AI agents are multiplying traffic overnight. Here's how to ship fast without taking your database down with it. Ben Dicken from PlanetScale argues you no longer have to choose between moving fast and keeping...
Continue reading
You may also like
Innohub TV
Watch this content also on Innohub TV
We picked clips from Innohub TV that continue the same topic.
