RAG in Microsoft Agent Framework Step By Step

Posted on October 1, 2026 by Jesse Liberty

Adding RAG to Microsoft Agent Framework is both straightforward and confusing. OK, that makes no sense, but I’ve been struggling to understand all the steps. Here’s a first approximation, more to come in future posts (and I have a video coming with Bruno Capuano and Jon Galloway on this very subject). There are also a bunch of RAG blog posts that I’ve already posted; you can find the list here.



TL;DR

  • Learn the simple C# steps to wire Retrieval-Augmented Generation (RAG) into Microsoft Agent Framework (MAF).
  • Includes a runnable, self-contained console stub that chunks documents, stores them in-memory with metadata, runs a demo similarity search, and returns answers plus source citations.
  • Swap the demo token-overlap search for real embeddings + a vector DB in production and then pass the search delegate into MAF’s TextSearchProvider.

Overview

  1. Chunk documents and record metadata (Id, SourceName, SourceLink).
  2. Compute embeddings for each chunk and store (vector DB or in-memory for prototype).
  3. Implement a search delegate: query → embedding → nearest neighbors → map to TextSearchResult-like results.
  4. Attach the search delegate to MAF via TextSearchProvider. Choose between injecting passages (BeforeAIInvoke) or exposing search as a callable tool (OnDemandFunctionCalling).
  5. Run the agent; show model response and the underlying source metadata for traceability.

Notes on packages and API names

  • The relevant MAF package is typically Microsoft.Agents.AI (check the exact package and namespace for your MAF version on NuGet and the MAF docs).
  • Typical using directives you might need:
    using Microsoft.Agents.AI;
    using Microsoft.Agents.AI.TextSearchProvider;
  • API names and enum names (e.g., TextSearchBehavior.BeforeAIInvoke) may vary by MAF version—verify your installed NuGet version and adjust namespaces accordingly.

Quick actionable steps

  1. Prepare documents
  • Split long content into chunks (500–1000 chars recommended) with some overlap (50–100 chars) to preserve context.
  • For each chunk, store:
    • Id (unique)
    • SourceName (file name)
    • SourceLink (URL)
    • Text (chunk text)
    • Optional metadata (section title, date)
  • Compute an embedding for each chunk (production) and store the embedding vector alongside metadata.
  1. Implement the search delegate
  • Signature MAF expects (canonical shape you should adapt to your MAF version):
    Func<string, CancellationToken, Task<IEnumerable<TextSearchProvider.TextSearchResult>>>
  • Implementation steps:
    • Convert query → embedding (same embedding model used for docs).
    • Query vector DB for top-K neighbors (approx. nearest neighbor).
    • Map each hit to TextSearchResult (Text, SourceName, SourceLink, RawRepresentation).
  1. Create and configure a TextSearchProvider
  • Instantiate a TextSearchProvider with your search delegate and options (behavior, top_k).
  • Add to ChatClientAgentOptions.AIContextProviders so the agent can use it.
  1. Run the agent and surface metadata
  • Decide the mode:
    • BeforeAIInvoke—passages injected into prompt before model call (simpler, higher token usage).
    • OnDemandFunctionCalling—model invokes search explicitly (more control/cost saving).
  • Return the model response, and surface SourceName/SourceLink/RawRepresentation to the user for verification.

Scoring and thresholds

  • Use cosine similarity on embeddings—normalized between -1 and 1. Typical positive ranges near 0.7–0.95 indicate high similarity depending on your model and vector dims.
  • Prototype token-overlap scores are uncalibrated. Use a minScore cutoff (e.g., 0.05–0.2) to filter noise and avoid returning unrelated passages.
  • Top-K: usually 3–5 passages works well to balance context and token cost.

Troubleshooting quick tips

  • If queries return nothing: check chunking (too small chunks can lose context) and tokenization differences (embedding model vs. tokenization used in your search).
  • If irrelevant results: increase overlap, reduce chunk size, or switch to a stronger embedding model.
  • If costs are high: use OnDemandFunctionCalling and caching of query embeddings.

Runnable, self-contained C# console stub

  • Purpose: demonstrate chunking, adding documents with metadata, a demo token-overlap similarity search, returning top-K hits, and showing source metadata.
  • This demo is intentionally self-contained and uses a simple token-overlap scoring (NOT production quality). Replace the demo search and Tokenize with real embeddings + vector DB.

Create a new console app:
dotnet new console -n SimpleRagDemo
Replace Program.cs with the code below and run dotnet run.

Code (runnable stub)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using System.Text;

// Simple Document chunk container with metadata
record DocumentChunk(string Id, string SourceName, string SourceLink, string Text);

// Result shape similar to TextSearchProvider.TextSearchResult
record SearchResult(string Text, string SourceName, string SourceLink, double Score, string RawRepresentation);

class SimpleRagStore
{
    private readonly List<DocumentChunk> _chunks = new();

    // Add a chunk (in production you'd also persist an embedding vector here)
    public void AddChunk(DocumentChunk chunk) => _chunks.Add(chunk);

    // Demo-only: token-overlap similarity (replace with embeddings + vector DB in production)
    // Returns topK search results (and ignores zero-score results by default)
    public Task<IEnumerable<SearchResult>> SearchAsync(string query, int topK = 3, double minScore = 0.0, CancellationToken ct = default)
    {
        ct.ThrowIfCancellationRequested();

        var qTokens = Tokenize(query);
        if (qTokens.Count == 0) return Task.FromResult(Enumerable.Empty<SearchResult>());

        var scored = _chunks.Select(c =>
        {
            var dTokens = Tokenize(c.Text);
            var overlap = qTokens.Intersect(dTokens).Count();
            // Simple normalized score for demo (not cosine)
            var score = overlap == 0 ? 0.0 : overlap / Math.Sqrt(qTokens.Count * dTokens.Count);
            return new SearchResult(
                Text: c.Text,
                SourceName: c.SourceName,
                SourceLink: c.SourceLink,
                Score: score,
                RawRepresentation: $"chunk:{c.Id}"
            );
        })
        .Where(r => r.Score > minScore)
        .OrderByDescending(r => r.Score)
        .Take(topK);

        return Task.FromResult(scored.AsEnumerable());
    }

    // Demo tokenization: extract word-like tokens, lower-case, remove short stop-words.
    // Production: keep consistent tokenization with embedding model or skip and use vector similarity.
    private static HashSet<string> Tokenize(string text)
    {
        var matches = Regex.Matches(text.ToLowerInvariant(), @"\w+"); // letters/digits/underscore
        var set = new HashSet<string>();
        foreach (Match m in matches)
        {
            var tok = m.Value;
            if (tok.Length <= 2) continue; // skip tiny tokens in this demo
            set.Add(tok);
        }
        return set;
    }
}

static class ChunkingHelper
{
    // Split a long document into chunks with overlap.
    // chunkSize characters and overlap characters.
    public static IEnumerable<string> ChunkText(string text, int chunkSize = 800, int overlap = 100)
    {
        if (string.IsNullOrEmpty(text)) yield break;
        if (chunkSize <= overlap) throw new ArgumentException("chunkSize must be greater than overlap");

        int start = 0;
        while (start < text.Length)
        {
            int len = Math.Min(chunkSize, text.Length - start);
            yield return text.Substring(start, len);
            start += (chunkSize - overlap);
        }
    }
}

class Program
{
    // Placeholder: in production you'd call your embedding model here.
    // Keep this method so it's clear where to swap in real embeddings.
    static Task<double[]> GetEmbeddingAsync(string text, CancellationToken ct = default)
    {
        // Demo does not use embeddings. Return a dummy vector for shape.
        return Task.FromResult(new double[0]);
    }

    static async Task Main(string[] args)
    {
        var store = new SimpleRagStore();

        // Example long document (will be chunked)
        var longDoc = @"This document explains how to reset your password. You can reset your password
by visiting /account/reset and following the instructions. Reset tokens last 1 hour.
If you don't receive an email, check your spam folder and contact support at support@example.com.";

        // 1) Chunk the long document and add chunks with metadata
        int chunkSize = 200; // small for demo visibility
        int overlap = 40;
        int chunkIndex = 0;
        foreach (var chunkText in ChunkingHelper.ChunkText(longDoc, chunkSize, overlap))
        {
            var id = $"pwd-doc-{chunkIndex++}";
            store.AddChunk(new DocumentChunk(
                Id: id,
                SourceName: "AccountFAQ.md",
                SourceLink: "https://example/docs/account-faq",
                Text: chunkText.Trim()
            ));
        }

        // Add a few short docs (already small)
        store.AddChunk(new DocumentChunk("install-1", "GettingStarted.md", "https://example/docs/getting-started",
            "To install the product, run dotnet tool install -g Example.Tool and then run example init."));
        store.AddChunk(new DocumentChunk("rn-1", "ReleaseNotes.md", "https://example/docs/releases",
            "Version 2.1 adds RAG support via a TextSearchProvider and includes an in-memory connector."));

        Console.WriteLine("Simple RAG demo. Ask a question about the docs (type 'exit' to quit).");

        while (true)
        {
            Console.Write("> ");
            var question = Console.ReadLine();
            if (string.IsNullOrWhiteSpace(question)) continue;
            if (question.Trim().Equals("exit", StringComparison.OrdinalIgnoreCase)) break;

            // 2) Search (this mimics the search delegate you'd pass into TextSearchProvider)
            var results = await store.SearchAsync(question, topK: 3, minScore: 0.0, CancellationToken.None);

            // 3) Synthesize a simple grounded answer (demo-only)
            if (!results.Any())
            {
                Console.WriteLine("No grounded passages found. (In production you might still call your LLM but mark as 'no sources found')");
                continue;
            }

            var best = results.First();
            var outBuilder = new StringBuilder();
            outBuilder.AppendLine("Answer (grounded):");
            outBuilder.AppendLine($" - {best.Text}");
            outBuilder.AppendLine();
            outBuilder.AppendLine("Sources:");
            foreach (var r in results)
            {
                outBuilder.AppendLine($" - {r.SourceName} ({r.SourceLink})  [score={r.Score:F3}] raw={r.RawRepresentation}");
            }

            Console.WriteLine(outBuilder.ToString());
        }
    }
}

How to swap the demo for real production components

  • Replace Tokenize + token-overlap similarity with:
    • Embeddings: call your embeddings model (Azure OpenAI or OpenAI), get float[] vectors for each chunk and for the query.
    • Vector store: persist chunk vectors + metadata to Qdrant, Redis, Azure Cognitive Search (vector-capable) or pgvector and query via nearest-neighbor search (cosine or dot-product).
    • Map DB hits to TextSearchProvider.TextSearchResult and return top-K.
  • Ensure consistent embedding model between document and query vectors.

Mapping the search delegate into Microsoft Agent Framework (sketch)

  • Search delegate signature (adapt to MAF version):
    Func<string, CancellationToken, Task<IEnumerable<TextSearchProvider.TextSearchResult>>>
  • Example wiring (pseudo):
// Adapt namespaces & types for your MAF version
async Task<IEnumerable<TextSearchProvider.TextSearchResult>> MySearchAsync(string q, CancellationToken ct)
{
    var qVec = await MyEmbeddingClient.GetEmbeddingAsync(q, ct);
    var hits = await MyVectorDb.SearchAsync(qVec, topK: 4, ct);
    return hits.Select(h => new TextSearchProvider.TextSearchResult
    {
        Text = h.Text,
        SourceName = h.Metadata["sourceName"],
        SourceLink = h.Metadata.TryGetValue("link", out var l) ? l : null,
        RawRepresentation = h.Id
    });
}

var textSearchProvider = new TextSearchProvider(MySearchAsync, new TextSearchProviderOptions
{
    // Behavior = TextSearchBehavior.BeforeAIInvoke or OnDemandFunctionCalling
});
var agentOptions = new ChatClientAgentOptions
{
    Name = "MyRagAgent",
    AIContextProviders = new[] { textSearchProvider }
};
  • Note: check MAF docs for exact type/namespace names for your NuGet version.

Practical tuning checklist

  • Chunk size: 500–1000 chars is common. Use overlaps of 50–150 chars.
  • Top-K: start with 3–5. Increase if model still hallucinates.
  • Min score: filter out low similarity hits.
  • Mode: BeforeAIInvoke for reliability; OnDemandFunctionCalling for cost control.

Security and cost considerations

  • Embeddings and model calls cost money—cache embeddings and reuse them.
  • Remove/obfuscate PII before storing embeddings if required by privacy rules.
  • Log and surface source metadata for auditability.

Sample console output (what to expect)

  • User: “How do I reset my password?”
  • Output:
    Answer (grounded):

Note: The initial draft of this post was written by BlogWriter and then edited by Jesse Liberty.
Illustrations by Copilot. Caution: LLMs make mistakes; this post is offered as is.

Unknown's avatar

About Jesse Liberty

Jesse Liberty has three decades of experience writing and delivering software projects and is the author of 2 dozen books and a couple dozen online courses. Liberty is a Senior AI Engineer at the University of Pittsburgh Medical Center, and was a Team Lead and Senior Software Engineer for various corporations, a Senior Technical Evangelist for Microsoft, a Distinguished Software Engineer for AT&T, a VP for Information Services for Citibank and a Software Architect for PBS. He is a 21 year Microsoft MVP.
This entry was posted in AI, Essentials. Bookmark the permalink.