When Is the LLM Called?

When working with agents (e.g., in Microsoft Agent Framework) it is easy to be confused as to when the LLM is called. Since calling the LLM (and getting responses) costs money (in the form of expended tokens) it is important to understand this relationship.

An agent in the Microsoft Agent Framework (MAF) talks to an LLM whenever it needs to:
(1) decide what to do or generate text (initial planning/response generation),
(2) decide whether to call a tool and provide its arguments (function/tool calling), and
(3) incorporate tool outputs and continue reasoning or synthesize the final reply.

A single agent run commonly produces multiple model calls in a loop: planning → tool call(s) → synthesis → (repeat if needed).

MAF implements a layered agent pipeline. Calling agent.RunAsync (or equivalent) starts a run that flows down middleware and context providers before ever hitting a model client (the LLM), and the run’s responses flow back up. That orchestration is why an agent doesn’t “just call the LLM once” by default—the framework determines when and how many model requests are required based on the agent’s instructions, tools, and middleware.

Three middleware layers you can hook in

• Agent-run middleware: wraps the entire run (fires once per run). Good for run-level telemetry or cancellation.
• Function/tool middleware: wraps each tool invocation (fires for each tool call). Useful for gating tool execution, approvals, or stubbing tools for tests.
• Chat / IChatClient middleware: wraps every underlying model/chat client call (fires every time the framework calls the model). Useful for logging prompts, redaction, or cost measurement.

Typical sequence that triggers LLM calls

1. Request received → pipeline and context providers populate run context (no model call yet).
2. Initial reasoning/plan → LLM call to decide plan (generate response or choose a tool). This is typically the first chat call.
3. If the model chooses a tool → framework executes function middleware and runs the tool (tool execution itself may be non-LLM code).
4. After tool returns → agent calls LLM again to integrate tool output and produce/synthesize the final response.
5. If needed, the LLM may ask to call another tool and the loop continues. Each cycle can add one (or more, depending on provider semantics) model calls.

C# examples

1. Minimal agent run (RunAsync typically triggers at least one LLM call)
   This follows the quick-start pattern; RunAsync goes through the pipeline and will call the model as needed.

using System;
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;

var agent = new AIProjectClient(
new Uri("https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project"),
new AzureCliCredential()
).AsAIAgent(
model: "gpt-5.4-mini",
instructions: "You are a helpful assistant. Keep answers brief."
);

var result = await agent.RunAsync("What is the largest city in France?");
Console.WriteLine(result);

2. Tool/function call flow (model decides to call a tool → tool executes → model synthesizes)
   Sketch of registering a tool; the run will typically perform model → tool → model:

// sketch
var myTool = new FunctionTool(
name: "get-weather",
description: "Get weather for a city",
handler: async (args) => {
return await GetWeatherAsync(args); // actual non-LLM work
}
);

var agent = new ChatClientAgentBuilder()
.UseModel("gpt-5.x")
.RegisterTool(myTool)
.Build();

// This RunAsync will likely:
// 1) call model to decide plan (may return "call get-weather")
// 2) execute myTool (no LLM)
// 3) call model to synthesize the final reply
var response = await agent.RunAsync("What's the weather in Seattle tomorrow?");
Console.WriteLine(response);

3. Chat middleware: intercept every LLM call (log each model invocation)
   Chat-level middleware runs on every model/chat client request:

async Task LoggingChatMiddleware(
IEnumerable messages,
ChatOptions? options,
IChatClient innerClient,
CancellationToken cancellationToken)
{
  Console.WriteLine($"[ChatMiddleware] Sending {messages.Count()} messages to model...");
  var resp = await innerClient.GetResponseAsync(messages, options, cancellationToken);
  Console.WriteLine($"[ChatMiddleware] Model returned {resp.Messages.Count} messages.");
  return resp;
}

// Register this middleware when configuring IChatClient / ChatClientAgent (see docs for exact APIs).

4. Agent-run and function middleware (brief sketches)

• Agent-run middleware: wrap whole run for telemetry or per-run constraints.
• Function middleware: intercept each tool invocation (e.g., require approval):

// Function middleware sketch
async Task ApproveThenExecuteMiddleware(
ToolInvocation invocation,
Func<ToolInvocation, Task> next,
CancellationToken ct)
{
  if (!IsApproved(invocation)) throw new Exception("Tool call not approved.");
    return await next(invocation); // proceed to actual tool handler
}

Practical caveats and best practices

• Multiple LLM calls per run are normal. Instrument chat middleware to measure API usage and cost.
• Tool execution is non-LLM work; function middleware can gate, mock, or log tool runs.
• Providers differ: function-calling semantics, streaming behavior, and limits can change how often or how responses are streamed.
• Use concise instructions and tool schemas so the model makes fewer unnecessary tool calls.
• If you need strict accounting of model usage, intercept at chat middleware (every model call passes there).