🔍 Your AI Application Is Broken—Here’s What to Do About It
It's data, data, data, all the way down!
Welcome back to Vanishing Gradients! This edition is all about the often-overlooked foundation of AI and ML: data. From debugging AI systems to iterating on prototypes and scaling ML applications, we’re diving into the data-driven realities behind successful AI—not just the models.
Here’s what’s inside:
📊 Data Intelligence > AI Hype: Ari Kaplan (Global Head of Evangelism, Databricks) on why your data pipeline—not your model—is your biggest advantage.
🔍 Looking at Your Data: Hamel Husain (parlance labs) on why evals alone won’t fix broken AI applications.
🛠 Rapid Iteration & Prototyping: A free workshop with Ravin Kumar (DeepMind) on refining AI applications through evaluation and debugging.
🎙 Conversations with Joe Reis & Ryan Stevens (Ramp): AI agents, automation, and the ML lifecycle from two industry leaders.⚡ Lightning Lesson: When (if ever) you should use LLM agents—and why they’re often overkill.
It’s data all the way down. Let’s delve dive in.
🛠 Free Workshop: Iterating on AI Applications with Ravin Kumar (DeepMind)
Building AI applications is easy—until you need them to actually work. That’s why Ravin Kumar (DeepMind) and I are running a free, 3-hour online workshop focused on taking AI apps beyond the prototype stage through evaluation, debugging, and iteration.
This workshop builds on Ravin’s guest lecture in my Building LLM Applications course, where he shared insights from his work at Google and DeepMind on AI product development. Now, we’re taking it a step further with a hands-on session where you’ll refine an AI app in real time.
What we’ll cover:
✅ Evaluation & tuning—iterating on prompts with evaluation sets
✅ First-pass optimization—speed, cost, and practical trade-offs
✅ Iteration & model selection—debugging failure cases and improving reliability
📅 Date: March 2nd, 7 PM PT
📍 Where: Online
This workshop is focused on improving AI applications through evaluation, debugging, and iteration—it’s not about large-scale infrastructure or high-throughput production systems. If you want to go beyond a basic AI demo and make your app more reliable and effective, this session is for you.
🔍 Your AI Application Is Broken—Here’s What to Do About It
Most teams debug AI applications the wrong way.
They assume the solution is to throw an evaluation library at the problem—but that doesn’t actually help them understand what’s broken. In the latest Vanishing Gradients episode, Hamel Husain pushes back on that assumption. If you’re relying on evals without actually looking at your data, you’re probably missing what’s really wrong.
Key takeaways:
🔍 Look at your data — You don’t need fancy tools to do this. A simple spreadsheet or text file is enough to start reviewing your model’s behavior.
✅ Define what "correct" means — AI outputs aren’t binary right or wrong; they depend on context. Work with domain experts to clarify what correctness actually looks like.
⚠️ Avoid the tool trap — Dashboards and evaluation tools don’t fix bad outputs. The first step is inspecting results and understanding failures.
🛠 Evaluations are essential — Start small and iterate. Even basic automated checks help prevent silent failures over time.
Matt Stockton wrote a great post expanding on this conversation—check out his blog post here: 🔗 Read it here.
🎥 Watch the clip to hear Hamel’s take:
🎧 Listen to the full episode:
🔗 Spotify
By the way, Hamel and Shreya Shankar (UC Berkeley, ex-Meta, Google Brain) have just launched their course Consistently Improve Any AI Application With Evals, starting in May. I’m planning to take it myself, and if you’d like to join me, you can use this coupon for $100 off.
💡 AI Won’t Save You—But Data Intelligence Might
In the latest High Signal podcast, I spoke with Ari Kaplan, Global Head of Evangelism at Databricks and a pioneer in sports analytics, about how data—not AI—drives real impact.
Ari has built analytics teams for Major League Baseball, advised McLaren’s F1 racing strategy, and helped top companies apply AI where it actually works—without getting lost in the hype.
What we covered:
⚾ How data (not better players) changed Major League Baseball—forcing rule changes.
📊 Why AI models are a commodity, but your data pipeline is where the real advantage lies.
🤖 How companies can use AI agents effectively—without running into their biggest failure modes.
🔍 Why improving retrieval and data quality often matters more than upgrading models.
In this clip, Ari breaks down how a simple data-driven strategy transformed baseball—and why every industry should take note.
🎥 Watch the clip to hear his take.
🎧 Listen to the full episode:
🔗 Spotify
📢 More AI & ML Updates
🔊 AI Agents & Automation on The Joe Reis Show. I joined Joe Reis on The Joe Reis Show to discuss the next wave of AI agents and automation—what’s real, what’s hype, and how these technologies are evolving.🎧 Listen here.
📣 SciPy 2025: Call for Proposals Extended! I’m co-chairing the Data Science, ML, and Explainable AI track at SciPy this year with Dr Fatma Tarlaci (Chief AI Officer, Soar.com) and Marlene Mhangami (Developer Advocate, Microsoft). If you’ve got something to share, we’d love to see your proposal!
📝 New CFP deadline: March 5🔗 Submit your proposal here
🔥 Fireside Chat: The ML Lifecycle with Ryan Stevens (Ramp). I hosted a fireside chat with Ryan Stevens (Head of Applied Science at Ramp, ex-Meta) on building scalable ML systems. We covered reproducibility, structured ML workflows, and how applied science teams evolve. 🎥 Watch here.
⚡ Upcoming Lightning Lesson: LLM Agents—When to Use Them (and When Not To)Stefan Krawczyk and I are teaching a free, 30-minute Lightning Lesson on when LLM agents actually make sense—and when they don’t.
We’ll break down:
✅ When to use LLM agents vs. simpler alternatives
🛠 How to structure agentic workflows with tool use
🔍 Debugging & evaluating agents for reliability & scalability
📅 Date: March 3, 2025
🕙 Time: 14PM PT
Where: Virtual (Zoom)
🔗 Register here (if you can’t make it, register and we’ll share the recording after)
Want to Support Vanishing Gradients?
If you’ve been enjoying Vanishing Gradients and want to support my work, here are a few ways to do so:
🎓 Cohort 2 of Building LLM Applications is now open! – The first run was a blast—over 80 students from Meta, Adobe, TikTok, Ford, and the US Air Force, and more joined. The Discord community has been incredibly active, and we’ve had guest speakers like Hamel Husain, Swyx, Ravin Kumar (DeepMind), Eric Ma (Moderna), and Xharmagne Carandang (Lorikeet). If you want to join the second cohort or think a colleague/friend would, check it out here (also hit reply to let me know if a group discount would be of interest!)
⚡ Join an upcoming Lightning Lesson – On March 4, Stefan Krawczyk and I are teaching LLM Agents: When to Use Them (and When Not To). If you’re curious about agentic workflows, debugging, and when simpler alternatives are better, register here.
🎙 Spread the word – If you find this newsletter valuable, share it with a friend, colleague, or your team. More thoughtful readers = better conversations.
📅 Stay in the loop – Subscribe to the Vanishing Gradients calendar on lu.ma to get notified about livestreams, workshops, and events.
▶️ Subscribe to the YouTube channel – Get full episodes, livestreams, and AI deep dives. Subscribe here.
💡 Work with me – I help teams navigate AI, data, and ML strategy. If your company needs guidance, feel free to reach out by hitting reply.
Thanks for reading Vanishing Gradients!
If you’re enjoying it, consider sharing it, dropping a comment, or giving it a like—it helps more people find it.
Until next time,
Hugo