Cutting AI Assistant Costs by Up to 77.8%: The Power of Enhancing LLMs with Business Logic

Kicking off Data Dialogs, a new series of data, ML, and AI conversations for data leaders

Hi all! I’ve recently started a newsletter around all things data science, ML, and AI, primarily to keep track of interesting things in the space and what I’ve been up to. This is an experiment so please do let me know what you’d like to see here. There’s a lot to share this week so let’s jump right in.

Kicking off Data Dialogs with Delphina

I’m excited to announce that I’m kicking off a new series of data, ML, and AI conversations for data leaders with my friends at Delphina. Many of you may know that I have a lot of public conversations in the space, but there are some things that people don’t want to chat about so publicly. And truth be told, there aren’t so many spaces for data leaders to have open, honest dialogs.

This is why we’re starting Data Dialogs, a fortnightly application-only online private forum where top minds in data science, ML, and AI can learn, connect, and tackle pressing challenges together.

First session:

• 🗓️ Thursday, August 22, 4:00 PM PT on Zoom

• 🎙️ Featured speaker: Brad Klingenberg, founder of Naro and former Chief Algorithms Officer at Stitch Fix

• 📌 Topic: "Navigating the AI Revolution: where's the data science?" — exploring the interplay between traditional ML and GenAI (the good, the bad, and the ugly) and how it's reshaping our field.

The Dialogs are a space for authentic conversations. No recordings, no sales pitches — Chatham House Rules and real talk about timely topics.

We’ve got a deep roster of incredible speakers to come, spanning all facets of data science strategy, tactics, and infrastructure including Lilei Xu, Min Cai, Kane Sweeney, Ali Rauh, Victor Kostyuk and more.

Check out the full details and apply to join us.

Cutting AI Assistant Costs by Up to 77.8%: The Power of Enhancing LLMs with Business Logic

🚀 We know LLMs are powerful in many ways, but it still isn't clear how to deploy them in production applications. This is a challenge many organizations are grappling with.

🤖 My friends at Rasa have been building Conversational AI tools for developers, data scientists, and MLEs in Fortune 500 companies for nearly a decade, and have developed some robust ways of embedding LLMs in conversational software.

📊 So Alan Nichol, Daksh Varshneya, and I recently conducted a study comparing CALM (Conversational AI with Language Models) with LangChain/LangGraph. Key findings were:

• 💰 Cost Efficiency: CALM reduced operational costs by up to 77.8%

• ⚡ Speed: 4x faster response times compared to LangChain/LangGraph

• 🎯 Reliability: Significantly higher consistency in following business rules

• 🔄 Flexibility: Maintained adaptability while ensuring adherence to crucial business logic

You can check out the full study here and I’m super interested if it’s valuable for you!

The hope was that we could offer valuable insights for anyone looking to implement LLM-powered AI assistants in real-world applications, balancing the power of LLMs with the need for consistent, reliable performance.

What We Learned Teaching LLMs to 1,000s of Data Scientists

I recently did a podcast with Dan Becker and Hamel Husain, two veterans in the world of data science, machine learning, and AI education. Collectively, they’ve worked at Google, DataRobot, Airbnb, Github (where Hamel built out the pre-cursor to copilot and more) and they both currently work as independent LLM and Generative AI consultants.

Dan and Hamel recently taught a course on fine-tuning large language models that evolved into a full-fledged conference, attracting over 2,000 participants.

In this episode, we dive deep into their experience and the unique insights it gave them into the current state and future of AI education and application. You can listen to the episode here or on your app of choice. You can also watch the livestream here:

Dan also did a live demo, which I’m pretty darn excited about, as it moves from the world of bits to atoms: it’s new pre-alpha stage application designed to help people create 3D printable physical objects using LLMs. You can check it out here and I’d love your thoughts:

The NLP and AI Revolution with spaCy Creators Ines Montani and Matthew Honnibal

I’ll be recording a Vanishing Gradients livestream with Ines and Matt from spaCy and Explosion. If you missed it in my last newsletter, I’m really pretty excited about this for many reasons. Here are a few:

• their work on NLP over the years is fertile ground for thinking through how to incorporate GenAI, ML, classic NLP, and software to build robust AI systems;

• their work in OSS has been inspirational for me in several regards, including how important UX and good abstraction layers are for developer tooling;

• we all have a lot to learn from their journey with respect to how OSS companies can be built and maintained sustainable.

Oh, and they’re also total legends! You can register for free here.

Also, if you’re interested in some of their more recent work, check out the following:

• Human-in-the-loop distillation: LLMs challenge industry workflows that need modularity, transparency and data privacy. But models don't have to be black boxes – you can distill them into better, smaller and faster components you can control and run in-house.

• How S&P Global is making markets more transparent with NLP, spaCy and Prodigy: Case study on real-time commodities trading insights using human-in-the-loop distillation.

• The AI Revolution Will Not Be Monopolized: Open source and interoperability means there's no monopoly to be gained in AI and economies of scale only matter if you buy into the "one model to rule them all" approach.

• Back to our roots: Ines and Matt are back to running Explosion as a smaller, independent-minded and self-sufficient company and focusing on our core stack, spaCy and Prodigy. I honestly think it’s wonderful that they’re sharing their learnings about building companies around OSS technologies. We’re all still learning so much.
  

I’ll be announcing more livestreams, events, and podcasts soon, so subscribe to the Vanishing Gradients lu.ma calendar to stay up to date. Also subscribe to our YouTube channel, where we livestream, if that’s your thing!

That’s it for now. Please let me know what you’d like to hear more of, what you’d like to hear less of, and any other ways I can make this newsletter more relevant for you,

Hugo