AI at NASA, Scaling Platforms at Uber, and the Future of Open-Source AI 🔭
From Prompt Engineering to AI Ethics: What’s Next for Generative AI?
Welcome back to Vanishing Gradients! I’ve recently started this newsletter to track all things data science, ML, and AI, as well as to share what I’ve been working on. This is still an experiment, so please let me know what you’d like to see more (or less!) of as we continue this journey.
Open Science at NASA -- Measuring Impact and the Future of AI
I recently sat down with Dr. Chelle Gentemann, Open Science Program Scientist at NASA, for a fantastic conversation about how NASA is integrating AI into its research lifecycle. We discussed everything from data accessibility to the challenges of open science. Chelle shared how NASA, IBM, and Clark University launched an open-source foundational model based on the Landsat dataset—available on Hugging Face!
Some highlights from the discussion:
🌍 Making NASA’s Data Accessible: Imagine asking NASA’s data about fire risks around your house—without needing a PhD!
🔬 Impact of Open Science: NASA is working to develop new metrics for measuring scientific impact, moving beyond publications to include contributions like software and data sharing:
🎙 Check out the full episode here (or on your app of choice) where we explore these topics in-depth.
📺 Watch the live stream here to see the full conversation!
Evaluating Chatbot Implementations: A Guide to Robust AI Agents
With the plethora of agents out there, I’m super interested in helping people figure out how to do robust evaluations of them. I’m excited to be working with the fine folk at Rasa, who’ve been building these systems for Fortune 100 companies for over a decade.
Some key takeaways from this work include:
🔑 Foundational Metrics: Always measure containment rate, customer satisfaction, and cost efficiency.
🚨 Avoiding Pitfalls: Watch out for common mistakes like inconsistent business logic or misinterpreted containment rates.
⚙️ Automation + Manual Review: Combine automated evaluation with manual oversight to ensure comprehensive results.
👉 Read the full post here to learn more about evaluating conversational AI systems effectively.
Building AI & ML Platforms – Lessons from Min Cai’s Journey at Uber
In our latest Data Dialogs session, Min Cai, custodian of Uber’s Michelangelo platform, shared valuable lessons on building scalable AI & ML platforms. We discussed feature engineering, large-scale model inference, and how Uber is integrating generative AI into its platform using a mix of GPT-4 and Llama models.
As this slide shows, Uber’s AI/ML journey can be categorized into three major phases:
2016–2019: Predictive ML — Uber started with tabular data use cases like ETA, risks, and pricing using XGBoost. This era also saw major investments in GPU scheduling and distributed training for deep learning, particularly for self-driving cars.
2019–2022: Deep Learning Expansion — By adopting deep learning and consolidating models in an ML monorepo, Uber experienced rapid growth in deep learning projects. More than 60% of Tier-1 models are now DL-based.
2023–2024: Generative AI Integration — Uber has begun integrating generative AI into its platform, using a mix of commercial models like GPT-4 and open-source solutions such as Llama models.
Key points from the session:
🚧 AI & ML Platforms: Build vs. Buy: The challenges Uber faced while scaling its platform.
🤖 Generative AI Integration: How Uber is unifying traditional ML and generative AI with a single API/UI.
📊 Platform Success Metrics: Including reliability, CSAT, and direct business impact.
Here’s a quick look at the Evolution of AI/ML @ Uber over the past few years:
For more, check out the full post I shared on LinkedIn. Also, don’t miss out on our next Data Dialog session with Victor Kostyuk, founder of OfferFit, where we’ll dive into applying self-learning AI and reinforcement learning to personalization and marketing. You can apply to join us here.
Community Spotlight: Automate the Boring Stuff with LlamaBot
Last week, I caught up with my friend Eric Ma (Senior Principal Data Scientist at Moderna Therapeutics). Eric showed me his latest project, LlamaBot, and I said let’s record a short demo!
This Python-based tool automates everyday tasks like generating commit messages and querying documents using LLMs, all within a developer-friendly workflow.
Some highlights:
Automated Commit Messages: LlamaBot auto-generates commit messages, complete with validation and even emojis. 🎉
Smart Text Validation: Ensures commit messages meet constraints, making your Git workflow more efficient.
LLM Integration: Connect to local models via Ollama or cloud-based LLMs like OpenAI.
🔗 Check out the GitHub repo and give it a star! 💫
Coming Up: Prompt Engineering and The Future of Open-Source AI and Research Infrastructure
I’ve got two livestreams coming up this week, which I’m excited about:
Prompt Engineering, Security in Generative AI, and the Future of AI Research
Are you ready to dive deep into one of the most effective techniques in generative AI? Join me, along with Sander Schulhoff (University of Maryland), Denis Peskoff (Princeton University), and Philip Resnik (University of Maryland), as we explore The Prompt Report—a comprehensive survey analyzing over 1,500 prompting papers. This session will give you practical insights into:
✏️ The evolution of prompt engineering and where it fits alongside fine-tuning and retrieval-augmented generation (RAG).
🔒 How to address security concerns like prompt injections and prevent vulnerabilities in generative AI systems.
📈 The future of AI research, including where prompt engineering might intersect with fields like robotics, AR, and VR.
Whether you’re interested in hands-on techniques or the larger research landscape, this session will offer valuable takeaways.
Sign up here and tune in for the conversation!
The Future of Open-Source AI and Research Infrastructure
Join me for a fireside chat with Hailey Schoelkopf, a leading researcher at EleutherAI—the team behind major open-source AI projects like GPT-NeoX. In this session, we’ll explore the evolving landscape of AI research and the critical role of open-source tools in advancing the field. Topics include:
💻 How EleutherAI went from a grassroots community to a leading nonprofit in AI research, driving open-source innovation.
📊 The impact of evaluation harnesses in AI research and industry adoption.
🌍 Challenges in building multilingual and multimodal AI models to serve global communities.
Join us to learn about the cutting edge of open-source AI.
Don’t miss this Outerbounds fireside chat—register here!
I’ll be announcing more livestreams, events, and podcasts soon, so subscribe to the Vanishing Gradients lu.ma calendar to stay up to date. Also subscribe to our YouTube channel, where we livestream, if that’s your thing!
That’s it for now. Please let me know what you’d like to hear more of, what you’d like to hear less of, and any other ways I can make this newsletter more relevant for you,
Hugo