Building Personal Agents with MCP

6 Engineering Principles for Building Reliable Agents

A detailed guide on developing scalable and reliable agentic workflows, taming agent behavior, and automating real-world tasks using MCP

In a recent lightning lesson for our Building with LLMs course, Skylar Payne, an AI leader with over a decade of experience at Google and LinkedIn, presented a masterclass on building custom AI tools.

He focused on moving beyond the hype of autonomous agents to the practical reality of creating reliable, scalable agentic workflows.

This post synthesizes his talk, offering a detailed guide on taming agent behavior, debugging complex systems, and automating real-world tasks using FastMCP.

All code is here.

All slides are here.

👉 Skylar and Hugo are running another free lightning lesson on October 10: Don’t Tweak Prompts. Engineer Agents. Register and/or get the recording here.👈

Enrolment is also open for our next cohort of Building LLM Applications for Data Scientists and Software Engineers (starting November 3). Use the coupon GENAI200OFF for $200 off.

Note: This post was written with Gemini (from the YT video and slides), using a variation of Hamel Husain’s notebook here.

Introduction

Skylar begins by setting the stage for the topics he’ll cover:

• building custom AI tools with FastMCP,

• debugging and guiding agent behavior, and

• developing scalable workflows to automate real-world tasks.

With his extensive background in scaling AI systems from startups to global platforms, he brings a pragmatic perspective, focusing on what actually works in production versus what often fails.

The Promise vs. Reality of AI Agents

The gap between agent hype and reality

The current narrative around AI agents is one of “boundless potential”. Skylar introduces the Model Context Protocol (MCP) as a key enabler of this vision. MCP aims to be a unified API that allows AIs to access tools and resources like Slack or Google Drive without needing custom integrations for each one. This unified interface should, in theory, allow an AI to perform complex tasks seamlessly.

Social media amplifies this promise with posts claiming to run dozens of autonomous agents with minimal oversight or create to-do lists that complete themselves.

However, the reality for most developers is often frustrating. You might ask an LLM to fix a bug, only for it to “solve” the problem by deleting the corresponding test case. This slide captures the stark contrast between the ambitious promise of AI agents and the often-unreliable reality of their current capabilities.

General Agents Are Not Generally Reliable

Data from the Berkeley Function Calling Leaderboard

To ground the discussion in data, Skylar presents the Berkeley Function-Calling Leaderboard, a benchmark that evaluates how well different models call the correct function at the right time. The results are sobering.

• Low Accuracy: The overall accuracy for even the best models doesn’t reach 90%. In the world of system reliability where success is measured in “nines” (e.g., 99.99%), not even having one nine is a major concern.

• Multi-Turn Degradation: Performance degrades significantly in multi-turn scenarios, where an agent must take multiple sequential actions to complete a task. This is precisely the kind of complex work we want to automate.

• Persistent Hallucinations: Models still exhibit high hallucination rates, which can introduce errors and unpredictability into any workflow.

• API Instability: An often-overlooked factor is the uptime and reliability of the third-party LLM APIs themselves. Latency spikes and provider outages can easily break an otherwise well-designed system.

These factors compound, creating a challenging environment for building automation that you can actually trust.

Workflows Over Agents: Subtracting Intelligence for Reliability

Omar Khattab (DSPy) on building reliable systems

Skylar highlights a powerful insight from Omar Khattab, the creator of Colbert and DSPy:

We don’t program software because we lack AGI. We program software because we want reliable systems.

The point is that we already have 8 billion examples of unreliable general intelligence (humans). The goal of engineering is to build predictable, reliable systems.

This often means subtracting intelligence in the right places. Instead of building a single, fully autonomous agent, the more robust approach is to create structured workflows composed of smaller, more predictable components. Agency should be reserved for small, focused parts of the system where it is truly needed, rather than being the default for the entire process. This “workflows over agents” philosophy is a central theme of the talk.

Why Building AI Agents Is Hard (And How to Fix It)

Challenges and principles for building agents

Building agentic systems presents several core challenges: iterations are slow due to API latencies, the internal reasoning of the LLM is a black box, systems quickly become complex, and everything is fundamentally unreliable.

To counter these difficulties, Skylar proposes a set of engineering principles that form the foundation of his approach:

1. Create a Fast Dev Loop: Make changes and see their effects instantly.

2. Strong Observability from the Start: Understand what the system is doing at all times.

3. Break Everything Down: Decompose large problems into small, composable tools and workflows.

4. Start Simple: Begin with basic tools and combine them into more complex workflows, which can then be exposed as tools themselves to an agent.

5. Use Agentic Coding: Leverage LLM assistants to help you build and test your tools.

6. Apply Reliability Techniques: Systematically use patterns to make unreliable components more robust.

The rest of the presentation demonstrates how to apply these principles in practice.

Demo: A Personal Weekly Review Agent

Skylar demonstrates a personal agent he built to help with his weekly review. He showcases two specific sub-tasks from a much larger workflow:

1. Email Triage: The agent reads unread emails, classifies them as either “needs reply” or “archive,” and bundles them for human approval. After approval, it archives the designated emails and drafts replies for the others.

2. LinkedIn Analytics: The agent uses browser automation to navigate to LinkedIn, log in, and scrape key analytics like follower count and impressions.

The demo highlights the human-in-the-loop pattern using a service called Human Layer, where Skylar approves or rejects the agent’s proposed email classifications in a simple UI before any action is taken. This ensures that a critical email is never accidentally archived.

The Tech Stack Under the Hood

Tools used to build the demo agent

The demo is powered by a handful of key tools:

• FastMCP: Implements the MCP server, allowing tools to be exposed over a network.

• Mirascope: A Python library for building LLM applications, used here for all AI functionality.

• Lilypad: Provides observability, tracing all function calls and logging them for debugging.

• Human Layer: A service for easily adding human-in-the-loop approval steps to any workflow.

Principle 1: Create a Fast Dev Loop

Code for enabling a fast development loop

A slow feedback cycle is a major productivity killer. To accelerate development, Skylar recommends two practices. First, instead of using the standard FastMCP command-line tool, run the server directly with uvicorn --reload. The --reload flag enables hot reloading, so the server automatically restarts whenever you save a change to the code. This provides instant feedback. Second, create a simple script to call your tools directly, allowing you to test individual components without running a full, multi-step agent.

Principle 2: Strong Observability from the Start

Observability with Lilypad and debugging with ipdb

Observability should be a day-zero priority, not an afterthought.

Skylar uses Lilypad, which makes tracing easy with a simple @trace decorator. A key feature of Lilypad is that it automatically versions your code, so when you review logs, you can see the exact state of the function that generated them. This is invaluable for debugging issues that appear over time. For interactive debugging, he recommends using ipdb. By adding import ipdb; ipdb.set_trace() to your code, you can pause execution and inspect variables, which works seamlessly with the hot-reloading server setup.

Principle 3: Build Small, Composable Bits

Example of tool composition

Composition is a fundamental software engineering principle that is crucial for building complex agentic systems. Instead of creating monolithic tools, build small, single-purpose tools that can be combined into more powerful workflows. The slide shows a simple example where a sum_of_squares tool is created by calling two simpler tools, add and square.

Skylar notes a current limitation of MCP: it lacks a schema for tool outputs, meaning the calling tool must manually parse the output of the tools it uses. Despite this “clunkiness,” the compositional approach is essential for managing complexity and creating reusable components.

Principle 4: Empower Your Agentic Coder

Using an LLM assistant to interact with a local MCP server

Modern LLM-based coding assistants like Claude are powerful development partners. Skylar recommends taking this a step further by installing your local MCP server and giving the assistant access to it. This allows the agent to not only write code for your tools but also to call and test them directly within the chat interface. This creates a highly interactive and efficient development workflow.

Anatomy of a Miroscope Agent

Code structure for a Miroscope agent using MCP

This slide presents the core structure of an agent built with Mirascope that uses MCP tools.

• The @llm.call decorator defines a function as an LLM prompt. The function’s body constructs the messages (system, user, history) sent to the model.

• A main run_agent function orchestrates the process. It connects to the MCP server, retrieves a list of available tools, and applies a filter to provide the agent with only the tools relevant to its current objective.

• The agent then runs in a loop, where each step can either be a tool call or the final answer. The loop continues until the task is complete or a maximum number of steps is reached.

Filtering the tools is a critical reliability pattern. The more tools an agent has access to, the more likely it is to choose the wrong one. By scoping its capabilities to the immediate task, you significantly improve its chances of success.

Additional Reliability Techniques

There wasn’t enough time to cover all the remaining slides on reliability but shared them as essential patterns for building robust systems.

Self-Correction: Have the agent review its own output. For example, after generating a plan, it can critique the plan against a set of predefined rules or a quality rubric and revise it before execution.

Human in the Loop: As seen in the demo, for any high-stakes action, pause execution and require explicit approval from a human before proceeding.

Validation & Retries: Use libraries like Pydantic to define an expected output schema and validate the LLM’s response. If validation fails or a transient network error occurs, automatically retry the call.

Fallbacks: If a powerful but less reliable model (like GPT-4) fails, have a simpler, more reliable fallback mechanism. This could be a smaller model, a deterministic rule-based system, or simply surfacing an error to the user.

Caching: Cache LLM responses to reduce latency and cost. For inputs that are likely to be repeated, returning a cached result is much faster and cheaper than making another API call.

---

👉 Skylar and Hugo are running another free lightning lesson on October 10: Don’t Tweak Prompts. Engineer Agents. Register and/or get the recording here.👈

Enrolment is also open for our next cohort of Building LLM Applications for Data Scientists and Software Engineers (starting November 3). Use the coupon GENAI200OFF for $200 off.

---

Q&A Session

Here are the questions from the audience and Skylar’s responses.

Q: Can this run as a complete background agent that doesn’t ask for feedback and learns which emails to archive on its own?

A: Yes, you can. To do this, you would first create a dataset. For example, you could query your Gmail for all emails without the “inbox” tag to get examples of archived messages, and query for emails you’ve replied to as positive examples. You can use this data to iterate on a prompt or agent and build an evaluation harness to measure its performance. You can then remove the human-in-the-loop step once you’re comfortable with its reliability and the risk of it making a mistake (e.g., archiving an important email).

Q: Can this be modified for LinkedIn engagement, like auto-commenting?

A: Yes, you could use browser automation to do this, but it’s a much harder task. The difficult part isn’t generating the comment; it’s reliably navigating to the correct post to comment on. You also need to be very careful. Most major platforms have anti-abuse systems that detect and ban accounts that use excessive automation.

Q: Where can we customize the prompts in this workflow?

A: This will be clear in the code shared in the GitHub repository. The prompts are defined within the functions that have the @llm.call decorator from Miroscope.

Q: What is your opinion on MCP? Is it reliable enough for production applications?

A: Probably not at the moment for most use cases. The biggest missing piece is security, particularly authentication. While you can mitigate this by running everything in a private subnet, the protocol needs a better security story before it’s truly production-ready for general use.

Q: What MCP protocol were you using? (STDIO, Server-Side Events, or HTTP)

A: I was using Server-Side Events (SSE). This was the original protocol for remote calls but is now considered legacy and has been deprecated in favor of the newer HTTP-based protocol.

Q: You mentioned tools should call other tools. Why compose them? Couldn’t that lead to code bloat and redundancy?

A: Composition is a cornerstone of good software engineering. The alternative to composition is rewriting everything from scratch for each new use case. Building small, reusable components and composing them into larger workflows is a proven way to manage complexity, just like the Unix philosophy of chaining simple commands with pipes. It’s about building a system of interoperable parts rather than monolithic applications.

---

Want to Support Vanishing Gradients?

If you’ve been enjoying Vanishing Gradients and want to support my work, here are a few ways to do so:

🧑‍🏫 Join (or share) my AI course – I’m excited to be teaching Building LLM Applications for Data Scientists and Software Engineers again in November with Stefan Krawczyk (Agentforce, Salesforce). If you or your team are working with LLMs and want to get hands-on, I’d love to have you.

📣 Spread the word – If you find this newsletter valuable, share it with a friend, colleague, or your team. More thoughtful readers = better conversations.

📅 Stay in the loop – Subscribe to the Vanishing Gradients calendar on lu.ma to get notified about livestreams, workshops, and events.

▶️ Subscribe to the YouTube channel – Get full episodes, livestreams, and AI deep dives. Subscribe here.

💡 Work with me – I help teams navigate AI, data, and ML strategy. If your company needs guidance, feel free to reach out by hitting reply.

Thanks for reading Vanishing Gradients! Subscribe for free to receive new posts and support my work.

Thanks for reading Vanishing Gradients! Subscribe for free to receive new posts and support my work.

If you’re enjoying it, consider sharing it, dropping a comment, or giving it a like—it helps more people find it.

Until next time ✌️

Hugo