Edit

Share via

Sample implementations of IChatClient and IEmbeddingGenerator

.NET libraries that provide clients for language models and services can provide implementations of the IChatClient and IEmbeddingGenerator<TInput,TEmbedding> interfaces. Any consumers of the interfaces are then able to interoperate seamlessly with these models and services via the abstractions.

The IChatClient interface

The IChatClient interface defines a client abstraction responsible for interacting with AI services that provide chat capabilities. It includes methods for sending and receiving messages with multi-modal content (such as text, images, and audio), either as a complete set or streamed incrementally. Additionally, it allows for retrieving strongly typed services provided by the client or its underlying services.

The following sample implements IChatClient to show the general structure.

using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

public sealed class SampleChatClient(Uri endpoint, string modelId)
    : IChatClient
{
    public ChatClientMetadata Metadata { get; } =
        new(nameof(SampleChatClient), endpoint, modelId);

    public async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> chatMessages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        // Simulate some operation.
        await Task.Delay(300, cancellationToken);

        // Return a sample chat completion response randomly.
        string[] responses =
        [
            "This is the first sample response.",
            "Here is another example of a response message.",
            "This is yet another response message."
        ];

        return new(new ChatMessage(
            ChatRole.Assistant,
            responses[Random.Shared.Next(responses.Length)]
            ));
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> chatMessages,
        ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        // Simulate streaming by yielding messages one by one.
        string[] words = ["This ", "is ", "the ", "response ", "for ", "the ", "request."];
        foreach (string word in words)
        {
            // Simulate some operation.
            await Task.Delay(100, cancellationToken);

            // Yield the next message in the response.
            yield return new ChatResponseUpdate(ChatRole.Assistant, word);
        }
    }

    public object? GetService(Type serviceType, object? serviceKey) => this;

    public TService? GetService<TService>(object? key = null)
        where TService : class => this as TService;

    void IDisposable.Dispose() { }
}

For more realistic, concrete implementations of IChatClient, see:

The IEmbeddingGenerator<TInput,TEmbedding> interface

The IEmbeddingGenerator<TInput,TEmbedding> interface represents a generic generator of embeddings. Here, TInput is the type of input values being embedded, and TEmbedding is the type of generated embedding, which inherits from the Embedding class.

The Embedding class serves as a base class for embeddings generated by an IEmbeddingGenerator<TInput,TEmbedding>. It's designed to store and manage the metadata and data associated with embeddings. Derived types, like Embedding<T>, provide the concrete embedding vector data. For example, an Embedding<float> exposes a ReadOnlyMemory<float> Vector { get; } property for access to its embedding data.

The IEmbeddingGenerator<TInput,TEmbedding> interface defines a method to asynchronously generate embeddings for a collection of input values, with optional configuration and cancellation support. It also provides metadata describing the generator and allows for the retrieval of strongly typed services that can be provided by the generator or its underlying services.

The following code shows how the SampleEmbeddingGenerator class implements the IEmbeddingGenerator<TInput,TEmbedding> interface. It has a primary constructor that accepts an endpoint and model ID, which are used to identify the generator. It also implements the GenerateAsync(IEnumerable<TInput>, EmbeddingGenerationOptions, CancellationToken) method to generate embeddings for a collection of input values.

using Microsoft.Extensions.AI;

public sealed class SampleEmbeddingGenerator(
    Uri endpoint, string modelId)
        : IEmbeddingGenerator<string, Embedding<float>>
{
    private readonly EmbeddingGeneratorMetadata _metadata =
        new("SampleEmbeddingGenerator", endpoint, modelId);

    public async Task<GeneratedEmbeddings<Embedding<float>>> GenerateAsync(
        IEnumerable<string> values,
        EmbeddingGenerationOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        // Simulate some async operation.
        await Task.Delay(100, cancellationToken);

        // Create random embeddings.
        return [.. from value in values
            select new Embedding<float>(
                Enumerable.Range(0, 384)
                .Select(_ => Random.Shared.NextSingle()).ToArray())];
    }

    public object? GetService(Type serviceType, object? serviceKey) =>
        serviceKey is not null
        ? null
        : serviceType == typeof(EmbeddingGeneratorMetadata)
            ? _metadata
            : serviceType?.IsInstanceOfType(this) is true
                ? this
                : null;

    void IDisposable.Dispose() { }
}

This sample implementation just generates random embedding vectors. For a more realistic, concrete implementation, see OpenTelemetryEmbeddingGenerator.cs.