As noted in a previous post, middleware plays a pivotal role in enhancing the functionality and observability of agents. The Microsoft Agent Framework utilizes two primary types of middleware: ChatClient Middleware and Agent Middleware. Understanding the distinctions between these two middleware types is essential for developers looking to optimize their agents’ performance and capabilities. This post will delve into the differences between ChatClient Middleware and Agent Middleware, illustrating their functionalities with examples, including a demonstration of function-invocation middleware for a single agent.
This is the approach I use in the demonstration program to log the invocation of the Tavily search tool.
What is Middleware?
To review, middleware is a software layer that acts as an intermediary between different software applications or components. In the context of the Microsoft Agent Framework, middleware enhances the interaction between agents and their underlying systems, allowing developers to intercept, modify, and log messages and operations. This capability is crucial for debugging, monitoring, and extending the functionality of agents.
ChatClient Middleware
Purpose
ChatClient Middleware is specifically designed to intercept calls made to an IChatClient implementation. This middleware is particularly useful for logging, modifying, or inspecting the raw messages exchanged between the agent and the underlying language model (LLM). By utilizing ChatClient Middleware, developers can gain insights into the communication flow, which is essential for debugging and improving the agent’s performance.
Use Case
A common use case for ChatClient Middleware is logging all messages sent and received by the agent. This can help developers understand how the agent interacts with users and the LLM, allowing for better optimization of responses and overall user experience.
Example
Here’s a simple example of how to implement ChatClient Middleware in C#:
var chatClient = new AIProjectClient(new Uri("your-uri"), new DefaultAzureCredential())
.GetProjectOpenAIClient()
.GetProjectResponsesClient()
.AsIChatClient(deploymentName);
var middlewareEnabledChatClient = chatClient
.AsBuilder()
.Use(getResponseFunc: CustomChatClientMiddleware, getStreamingResponseFunc: null)
.Build();In this example, CustomChatClientMiddleware would be a function that processes the messages before they are sent to or after they are received from the LLM. This middleware can log the messages, modify them, or even implement additional logic based on the content of the messages.
Agent Middleware
Purpose
Agent Middleware operates at a higher level than ChatClient Middleware. It allows for the interception of all agent runs, enabling developers to inspect and modify the input and output of the agent’s operations. This middleware is essential for managing the overall behavior of the agent, including session management, identity tracking, and token budget management.
Use Case
A typical use case for Agent Middleware is collecting information about the agent’s session or identity. For instance, if an agent needs to maintain context across multiple interactions or manage its resource usage effectively, Agent Middleware would be the appropriate choice.
Example
Here’s how you might implement Agent Middleware in C#:
var agent = new ChatClientAgent(middlewareEnabledChatClient, instructions: "You are a helpful assistant.");In this example, the ChatClientAgent is initialized with the middleware-enabled chat client. The agent can now leverage the capabilities of both ChatClient and Agent Middleware to enhance its functionality.
Function-Invocation Middleware for a Single Agent
Function-invocation middleware can be applied to both ChatClient and Agent Middleware. This type of middleware allows for the interception of function calls executed by the agent, enabling developers to inspect and modify inputs and outputs. This capability is particularly useful for logging, debugging, and implementing additional logic based on the agent’s operations.
Example of Function-Invocation Middleware in C
Here’s a simple example of how to implement function-invocation middleware for a single agent:
public class FunctionInvocationMiddleware
{
public async Task InvokeAsync(AgentRunContext context, Func<AgentRunContext, Task> next)
{
// Before the function call
Console.WriteLine($"Before function call: {context.FunctionName}");
// Call the next middleware in the pipeline
await next(context);
// After the function call
Console.WriteLine($"After function call: {context.Result}");
}
}
// Usage
var agent = new ChatClientAgent(middlewareEnabledChatClient, instructions: "You are a helpful assistant.")
.AsBuilder()
.Use(FunctionInvocationMiddleware.InvokeAsync)
.Build();In this example, the FunctionInvocationMiddleware class defines an InvokeAsync method that logs the function name before and after the function call. This middleware can be used to track the execution flow of the agent’s operations, providing valuable insights into its behavior.
In the demonstration program we modify the creation of the ResearcherAgent to ad this bit of code:
.Use(async (agent, context, next, cancellationToken) =>
{
if (context.Function.Name == tavilyTool.Name)
{
_logger.LogInformation(
"Researcher invoking Tavily tool '{Tool}' with arguments {Arguments}",
context.Function.Name,
context.Arguments);
}
return await next(context, cancellationToken);
})That’s the only change necessary to have the middleware capture the calls to Tavily tool by the Researcher agent. None of the other agents change.
The trace from this change looks like this:
[trace] → execute_tool tavily_search
info: BlogWriter.ResearcherAgent[0]
Researcher invoking Tavily tool 'tavily_search' with arguments [query, ChatClient middleware vs Agent middleware Microsoft Agent Framework]
[trace] ← execute_tool tavily_search (1796 ms)Summary
In summary, the Microsoft Agent Framework offers two distinct types of middleware: ChatClient Middleware and Agent Middleware.
- ChatClient Middleware focuses on the interaction with the chat client, allowing for the logging and modification of messages exchanged with the LLM.
- Agent Middleware deals with the overall operations of the agent, enabling developers to manage session information, identity, and resource usage.
Additionally, function-invocation middleware can be implemented to intercept function calls, allowing for detailed control over the agent’s behavior.
