Rethinking Data Science, ML, and AI

Systematically Improving LLM and RAG applications

Hi all! I’ve recently started a newsletter around all things data science, ML, and AI, primarily to keep track of interesting things in the space and what I’ve been up to. This is an experiment so please do let me know what you’d like to see here. There’s a lot to share this week so let’s jump right in.

Rethinking Data Science, ML, and AI

I recently did a podcast with Vincent Warmerdam, a senior data professional and machine learning engineer at :probabl, the exclusive brand operator of scikit-learn. Vincent is known for challenging common assumptions and exploring innovative approaches in data science and machine learning.

You can listen to the episode here or on your app of choice. You can also watch the livestream here:

Vincent is, among other things, a master storyteller with the ability to tell tales that communicate serious issues with how we work in the data world.

Here’s one such example from the podcast in which reframing a problem led to dramatic improvements for the World Food Organization:

Vincent also did some great demos, including

• Archive Front Page Project: A GitHub repository using actions for daily article scraping from arXiv, with classifiers to detect topics of interest.

• Neural Search for arXiv Papers: Using sentence transformers and Matrushka embeddings to search 600k CS abstracts, with prompting techniques and semi-supervised learning.

• Rethinking Scikit-learn Pipelines: Introduction to the "Playtime" library, offering a new syntax for declaring pipelines and simplifying feature engineering.

Check them out here:

Systematically Improving LLM and RAG applications

Dan Becker and Jason Liu are two of the people at the front-line of building and advising on enterprise-grade AI systems right now and they’re teaching a course on Systematically Improving LLM and RAG applications. Dan and Hamel Husain’s Mastering LLM course/conference was such a success, I’m interested to see how this one goes.

I recently had Jason on the Vanishing Gradients podcast (also see bottom of this email for a livestream I’m doing with Dan Becker soon!). You can check the whole conversation here but also take a peek at Jason’s RAG playbook in his consulting work:

If you’re interested in learning how to follow a repeatable process to continually evaluate and improve your RAG application, this course is for you!

If you’d like to see if the content will resonate with you, you can check out much of the content from the 1st course here.

From the Community: Reproducible Scientific Workflows

If you’ve been following this newsletter for a while, you may recall that I recently did a Fireside Chat with Wolf Vollprecht, creator of mamba, pixi (and more!)

I’ve started doing some work with Wolf and his team. They are doing so much fun stuff with AI and scientific package management at the moment. Check out what they’re up to in order to help scientists build more reproducible workflows with …. Pixi PDF!

Imagine embedding your ENTIRE dev environment IN A PDF!

• Ship code with your paper

• Instant reproducibility

• One-click rerun of analyses

Check out this short demo:

I’m interested in your thoughts on this, along with this post Wolf recently wrote on reproducibility challenges in scientific computing and Pixi's approach to solving them.

Building Reliable, Robust, and Custom ML and AI pipelines

OK, so a couple of days I livestreamed with Shreya Shankar (UC Berkeley, ex-Viaduct, Google Brain, and Facebook). Although we’d originally planned to focus on Generative AI Systems evaluation and how LLMs can scale human expertise, we ended up having a far more expansive conversation about

A vision of people being able to write bespoke/custom AI pipelines that are reliable and how you need

• good algorithms/data management to find and track good constraints/assertions (spade),

• good human in the loop processes to make sure your definition of reliable aligns with a systematic approach to evaluating constraints (evalgen),

• and a commitment to a flywheel where you are continually improving the system

You can check out the livestream above or wait for us to launch the podcast episode in the next week or so!

What’s on deck

If that’s not enough, later this month, I’ll be doing a Fireside Chat with Chip Huyen and you can register for free here.

I’m also doing a Vanishing Gradients livestream early August with Dan Becker and Hamel Husain about what they learned teaching AI, LLMs, and Fine-Tuning to 1,000s of data scientists and MLEs. You can register for free here.

I’ll be announcing more livestreams, events, and podcasts soon, so subscribe to the Vanishing Gradients lu.ma calendar to stay up to date. Also subscribe to our YouTube channel, where we livestream, if that’s your thing!

That’s it for now. Please let me know what you’d like to hear more of, what you’d like to hear less of, and any other ways I can make this newsletter more relevant for you,

Hugo