Running Microsoft Agent Framework Locally

Run a full Microsoft Agent Framework (MAF) agent entirely on-device to eliminate round trips to the cloud, reduce latency, avoid egress costs, and keep sensitive data private. With Foundry Local you can run optimized local model variants (ONNX/WinML) and wire them directly into MAF.

In this guide you’ll get a working quickstart (Windows .NET 7 + WinML and a cross-platform Python example), exact commands to install and download a model, copy‑and‑paste code, hardware/driver tips, observability and troubleshooting checks, CI/CD advice, and production‑ready recommendations.

What you’ll accomplish

• Install Foundry Local (runtime + SDK) and pick/download a local model from the Foundry catalog.
• Create a minimal MAF agent that uses Foundry Local as the model provider.
• Run the agent locally in C# (.NET 7 on Windows using WinML) and in Python (cross-platform).
• Diagnose common problems (OOM, slow inference, missing model) and tune model choices.

Quick overview (high-level steps)

1. Install Foundry Local SDK/CLI for your OS.
2. Use the Foundry CLI or SDK to list and download a model alias.
3. Add MAF + Foundry provider packages to your project (dotnet or pip).
4. Configure the client (FOUNDRY_LOCAL_MODEL env var or pass at construction).
5. Run the agent — Foundry Local will load the optimized model and MAF will call it for inference.

Prerequisites

• OS: Windows, macOS, or Linux
• .NET SDK: dotnet 7+ (if using C#)
• Python: 3.8+ (if using Python)
• Disk space: model downloads vary (see catalog); first download requires internet
• Hardware: CPU-only works, but GPU/NPU provides much faster inference. Ensure GPU drivers & runtimes are installed (see GPU section below).
• Command-line familiarity, ability to set environment variables

Quickstart (10–15 commands)
This is a condensed path from zero to a running agent (Windows example for .NET then Python). If you prefer a single OS, follow that section below in full.

Windows (.NET 7 + WinML quick path)

1. Install dotnet 7 SDK (if not installed).
2. Create a new console app and add packages:
   • dotnet new console -n FoundryAgentDemo
   • cd FoundryAgentDemo
   • dotnet add package Microsoft.Agents.AI
   • dotnet add package Microsoft.Agents.AI.Foundry –prerelease
   • dotnet add package Microsoft.AI.Foundry.Local.WinML
3. Set model alias (PowerShell):
   • $env:FOUNDRY_LOCAL_MODEL = “phi-4-mini”
4. Build & run:
   • dotnet build
   • dotnet run
     (On first run Foundry Local will download the chosen model. Expect time for the initial download.)

Cross-platform Python quick path

1. Create virtualenv and install (adjust package names if necessary — see details below):
   • python -m venv venv && source venv/bin/activate
   • pip install agent-framework foundry-local-sdk
2. Set model alias (bash):
   • export FOUNDRY_LOCAL_MODEL=”phi-4-mini”
3. Run main.py (see Python example below)

Important note: package and CLI names evolve. If a command fails, consult the Foundry Local docs and the official sample repos (foundry-samples and agent-framework) which contain tested examples and exact package/namespace names.

Part A — Install Foundry Local and download a model (details)

1. Install options
   Foundry Local can be consumed via:

• NuGet packages for .NET (recommended for C# apps)
• PyPI packages (for Python)
• npm packages (for JS)
• Native installers or a platform-specific CLI where available

Where to get the right installer/packages

• Microsoft Learn Foundry Local get-started (follow the platform-specific instructions)
• Official GitHub samples: microsoft-foundry/foundry-samples and microsoft/agent-framework

2. Example .NET installation (project-level)
   You don’t usually “install” the runtime globally for .NET; add the Foundry Local package(s) to your project:

• dotnet add package Microsoft.AI.Foundry.Local (Linux/macOS/CPU/GPU)
• On Windows to use WinML/DirectML acceleration:
  • dotnet add package Microsoft.AI.Foundry.Local.WinML

3. Example Python installation

• pip install foundry-local-sdk
• pip install agent-framework
  (Exact pip package names may vary across releases; the sample repo shows exact names. If pip cannot find the package, clone/from-source in the foundry-samples repo.)

4. The Foundry CLI (catalog management)
   Foundry exposes a catalog of local model aliases and (often) a CLI for listing/downloading models. CLI names vary by release (examples you may see in docs: foundryctl, foundry). Typical CLI flow:

• foundryctl catalog list
• foundryctl catalog inspect phi-4-mini
• foundryctl model download phi-4-mini –path /path/to/foundry/models

If your platform’s docs show a different CLI name, use that. The initial model download can be large; download location and cache are configurable in the runtime docs.

5. Choosing a model

• Check alias metadata for VRAM/disk estimates and license.
• If you have limited VRAM, pick smaller or quantized variants.
• Common small aliases: phi-4-mini, phi-2-mini, qwen-7b-quant (names change — check the catalog).

Part B — Copy‑paste ready C# example (Windows .NET 7 + WinML)

Important: package names and API surfaces can change between SDK releases. The commands below use package names commonly seen in the Microsoft samples. If NuGet packages change, run the dotnet add package commands without versions to get the latest packages, and refer to the foundry-samples repo for exact Program.cs if you encounter API differences.

1. Project setup (commands)

• dotnet new console -n FoundryAgentWinML
• cd FoundryAgentWinML
• dotnet add package Microsoft.Agents.AI
• dotnet add package Microsoft.Agents.AI.Foundry –prerelease
• dotnet add package Microsoft.AI.Foundry.Local.WinML
• (optional) dotnet add package Microsoft.Extensions.Hosting

2. Example Program.cs (copy into Program.cs)
   (This example shows the pattern: create Foundry local client, wrap in MAF chat agent, and run a prompt. Adjust namespaces if your SDK version uses slightly different names.)

using System;
using System.Threading.Tasks;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;

class Program
{
static async Task Main(string[] args)
{
// Read model alias from environment or use a default
var modelAlias = Environment.GetEnvironmentVariable(“FOUNDRY_LOCAL_MODEL”) ?? “phi-4-mini”;

    Console.WriteLine($"Using Foundry local model: {modelAlias}");

    // Create a FoundryLocalClient (provider glue). Exact constructor may vary by SDK version.
    var foundryClient = new FoundryLocalClient(model: modelAlias);

    // Create agent wrapper using a ChatClientAgent helper (example API)
    var agentOptions = new ChatClientAgentOptions
    {
        SystemPrompt = "You are a helpful assistant that summarizes text concisely."
    };

    var agent = ChatClientAgent.FromChatClient(foundryClient, agentOptions);

    var input = "Summarize the plan for the project in one sentence.";
    Console.WriteLine($"Sending prompt: {input}");

    var result = await agent.RunAsync(input);
    Console.WriteLine("Agent response:");
    Console.WriteLine(result);
}
}

3. Build and run

• Open PowerShell and set the model alias:
  • $env:FOUNDRY_LOCAL_MODEL = “phi-4-mini”
• dotnet build
• dotnet run

Notes

• On first run, Foundry Local will download and prepare model artifacts. Check the console logs for model download progress.
• If you want to pass the model explicitly instead of using an env var, call the FoundryLocalClient constructor with the desired alias (see SDK docs).

Part C — Copy‑paste ready Python example (cross-platform)

1. Create virtual environment and install (adjust if package names differ)

• python -m venv venv
• source venv/bin/activate (macOS/Linux)
• venv\Scripts\Activate.ps1 (Windows PowerShell)
• pip install agent-framework foundry-local-sdk

If PyPI names differ or packages aren’t present, clone the sample repo and install from source.

2. requirements.txt (example)
   agent-framework
   foundry-local-sdk
3. main.py (copy into your project)
   import os
   import asyncio

from agent_framework import Agent
from agent_framework.foundry import FoundryLocalClient

async def main():
model_alias = os.environ.get(“FOUNDRY_LOCAL_MODEL”, “phi-4-mini”)
print(f”Using Foundry local model: {model_alias}”)

# Create Foundry client (constructor may vary)
client = FoundryLocalClient(model=model_alias)

# Create a simple agent wrapper — adjust arguments to match the library you installed
agent = Agent(chat_client=client, instructions="You are a brief assistant that summarizes text.")

response = await agent.run("Explain Docker in one sentence.")
print("Agent response:", response)

if name == “main“:
asyncio.run(main())

4. Run

• export FOUNDRY_LOCAL_MODEL=”phi-4-mini” (macOS/Linux)
• $env:FOUNDRY_LOCAL_MODEL = “phi-4-mini” (Windows PowerShell)
• python main.py

Notes

• Use the sample repo’s Python examples to get exact import paths if the package you installed exposes different modules.
• On first run the Foundry runtime will download the model. Monitor logs.

Part D — Configure Foundry Local without env var (programmatic control)
Instead of environment variables, pass the model alias when you create the client:

• C#:
  var foundryClient = new FoundryLocalClient(model: “qwen2.5-1.5b-instruct”);
• Python:
  client = FoundryLocalClient(model=”qwen2.5-1.5b-instruct”)

This is useful for runtime selection, multi-agent scenarios, or when you want config inside your app.

Hardware, drivers and model sizing (practical guidance)

• NVIDIA GPUs: install NVIDIA drivers + CUDA toolkit compatible with the Foundry runtime. Many ONNX/WinML optimizations target CUDA 11.x; check Foundry docs for the exact CUDA/cuDNN versions required for a given release. Verify installation with:
  • nvidia-smi
• AMD GPUs: ROCm support varies by OS and GPU generation. Check ROCm compatibility guide.
• Windows DirectML / WinML: WinML provides DirectML acceleration; keep Windows and GPU drivers up-to-date. For Intel GPUs, install the latest Graphics drivers.
• VRAM and disk: model sizes and VRAM requirements are model-specific. Smaller models and quantized variants (4-bit/8-bit) are recommended for constrained devices. Use the Foundry catalog to inspect model metadata (size and suggested device types).
• If GPU memory is insufficient, pick (a) a smaller model, (b) a quantized variant, or (c) fall back to CPU execution (slower).

What to look for in the Foundry catalog

• Alias name
• Disk size and estimated VRAM
• Quantization variants offered (4-bit/8-bit)
• License and usage restrictions

Observability and logging

• Enable MAF OpenTelemetry integration to capture agent spans (system prompt, tool calls, model inference).
• Foundry Local logs (model download, provider selection, errors) appear in the runtime output. If you start Foundry as a background service, logs typically go to console files or a configured log directory — check the Foundry docs for the exact log file location on your OS.
• Example telemetry flow (conceptual):
  • Start tracing in your app (OpenTelemetry SDK).
  • Agent spans: pre-processing (prompt assembly), model inference (Foundry span), post-processing (tool calls).
  • Use traces to identify slow stages (model load vs. token-by-token decoding).

Troubleshooting checklist (concrete remedies)

1. Model not found / invalid alias

• Confirm alias with the Foundry catalog CLI:
  • foundryctl catalog list
  • foundryctl catalog inspect
• Ensure FOUNDRY_LOCAL_MODEL is set correctly.

2. First run stalls on download

• Check network connectivity and disk space.
• Watch Foundry log lines showing download progress.
• Consider pre-downloading models on CI machines to avoid repeated downloads.

3. Slow inference

• Confirm GPU provider selection: check Foundry logs for “selected execution provider” lines.
• Ensure GPU drivers and runtimes are properly installed (nvidia-smi, rocminfo).
• Use a smaller or quantized model if GPU not available.

4. Out-of-memory / VRAM errors

• Use smaller model alias or quantized variant.
• Use CPU execution for lower memory devices.
• If running multiple models concurrently, reduce concurrency or run only one model at a time.

5. API/namespace mismatches

• If compilation fails due to missing types/namespaces, check the sample repo for the exact package and API names for your SDK version. The foundry-samples and agent-framework samples are the definitive, copy‑paste-ready references.

6. Tooling and hosted features unavailable

• Foundry Local is a local chat client. Some hosted Foundry tools (hosted web search, hosted code interpreter) are not available locally. Build local replacements or use Foundry hosted services when you need managed tools.

Forensics & logs: where to look

• Application logs (your process) — model client creation and errors.
• Foundry runtime logs — download progress, execution provider selection.
• OS and GPU driver logs (nvidia-smi, dxdiag on Windows).
• CI logs for model download step (cache model artifacts to speed repeated runs).

CI/CD and production considerations

• Avoid repeated large downloads in ephemeral CI runners by caching the model artifacts. Use a build cache or artifact store with pre-downloaded model files.
• For smoke tests, use small local models or mocked chat clients to validate agent logic without heavy downloads.
• Memory budgeting: if multiple agents or other services use the same GPU, coordinate startup to avoid OOM.
• Hybrid deployment: run smaller on-device models for latency/privacy, and use hosted Foundry for heavy tooling or when you need managed, up-to-date models and scalable inference.

Security and licensing checklist

• Check model license in the Foundry catalog — confirm commercial usage terms if applicable.
• Store no secrets in code. Foundry Local does not require cloud API keys for local inference, but MAF or other integrations might — use secure stores.
• Limit permissions on model cache directories and follow OS best practices for process isolation.

Performance tuning tips

• Batch inference when possible (if the agent supports multiple concurrent requests).
• Use quantized model variants for lower memory and faster CPU inference; be conscious of potential accuracy trade-offs.
• Prefer GPU acceleration when available — it typically reduces latency per request significantly.
• Warm-up the model after download by running a small inference to optimize runtime caches.

Appendix: Useful command snippets (summary)

Set env var

• Windows PowerShell:
  • $env:FOUNDRY_LOCAL_MODEL = “phi-4-mini”
• macOS / Linux:
  • export FOUNDRY_LOCAL_MODEL=”phi-4-mini”

Dotnet project (example)

• dotnet new console -n FoundryAgentDemo
• dotnet add package Microsoft.Agents.AI
• dotnet add package Microsoft.Agents.AI.Foundry –prerelease
• dotnet add package Microsoft.AI.Foundry.Local
• dotnet run

Python virtualenv

• python -m venv venv && source venv/bin/activate
• pip install -r requirements.txt
• python main.py

Foundry CLI (example patterns — CLI name may vary)

• foundryctl catalog list
• foundryctl catalog inspect phi-4-mini
• foundryctl model download phi-4-mini –path /path/to/models

Where to get canonical examples

• Foundry-samples on GitHub: microsoft-foundry/foundry-samples (C# and platform-specific samples)
• Agent Framework repo: microsoft/agent-framework (sample projects for C# and Python)
• Microsoft Learn Foundry Local docs: (Foundry Local get-started & SDK reference)
  If any command or API fails, consult those repos for exact, copy-paste-ready Program.cs and main.py.

Foundry Local + MAF gives you an approachable path to local agents that respect privacy and latency constraints. The main friction is model selection and the first-time download/optimization. Start with the official sample repos (foundry-samples and agent-framework) for exact API versions, try a small model (phi-4-mini) for initial experiments, and iterate to larger/quantized models as your hardware allows

Note: The initial draft of this post was written by BlogWriter and then edited by Jesse Liberty
Illustrations by Copilot. Caution: LLMs make mistakes; this post is offered as is.