OMX Helsinki — S&P 500 — DAX — NASDAQ 100 — STOXX 600 — EUR/USD — EUR/SEK — BTC/USD — ETH/USD — Euribor 3M — Euribor 12M —
AI Engineer

AI Engineer

Technology and AI 1,040,000 subscribers

RAG Evaluation Is Broken! Here’s Why (And How to Fix It) – Yuval Belfer and Niv Granot

03.06.2025 1,063 views
Published: 03.06.2025 Category: Technology and AI Views: 1,063

Episode details

Episode description

Optimizing local benchmarks, chunking strategies, perfect retrieval scores. If you just nodded along, you're one of many developers building RAG systems optimized for metrics that don't matter in the real world. But what if our entire approach to evaluating retrieval-augmented generation is fundamentally flawed? The uncomfortable truth is that current RAG benchmarks reward systems that fail spectacularly on realistic information retrieval tasks. In this talk, I'll expose the critical gaps in how we evaluate RAG systems today, from the chunking catch-22 to the myth of perfectly contained information. Using examples like the "Seinfeld Test," we'll explore why high benchmark scores often lead to disappointed users. You'll learn practical strategies for meaningful RAG evaluation that reflects how information actually works in the wild, helping you build systems that impress not just benchmark leaderboards, but actual humans. To learn more, check out the full episode on RAG evaluation on YAAP: https://youtu.be/RsSkwpTmn8o?si=9gIR6EeIzPgbqY4O

Open the AI assistant chat. The chat loads only when you open it.