Trace AI applications using OpenAI SDK

2025-06-16

Tracing provides deep visibility into execution of your application by capturing detailed telemetry at each execution step. Such helps diagnose issues and enhance performance by identifying problems such as inaccurate tool calls, misleading prompts, high latency, low-quality evaluation scores, and more.

This article explains how to implement tracing for AI applications using OpenAI SDK with OpenTelemetry in Azure AI Foundry.

Prerequisites

You need the following to complete this tutorial:

An Azure AI Foundry project created.
An AI application that uses OpenAI SDK to make calls to models hosted in Azure AI Foundry.

Enable tracing in your project

Azure AI Foundry stores traces in Azure Application Insight resources using OpenTelemetry. By default, new Azure AI Foundry resources don't provision these resources. You can connect projects to an existing Azure Application Insights resource or create a new one from within the project. You do such configuration once per each Azure AI Foundry resource.

The following steps show how to configure your resource:

Go to Azure AI Foundry portal and navigate to your project.
On the side navigation bar, select Tracing.
If an Azure Application Insights resource isn't associated with your Azure AI Foundry resource, associate one.
To reuse an existing Azure Application Insights, use the drop-down Application Insights resource name to locate the resource and select Connect.

Tip

To connect to an existing Azure Application Insights, you need at least contributor access to the Azure AI Foundry resource (or Hub).
To connect to a new Azure Application Insights resource, select the option Create new.
1. Use the configuration wizard to configure the new resource's name.
2. By default, the new resource is created in the same resource group where the Azure AI Foundry resource was created. Use the Advance settings option to configure a different resource group or subscription.
  
  Tip
  
  To create a new Azure Application Insight resource, you also need contributor role to the resource group you selected (or the default one).
3. Select Create to create the resource and connect it to the Azure AI Foundry resource.
Once the connection is configured, you are ready to use tracing in any project within the resource.
Go to the landing page of your project and copy the project's endpoint URI. You need it later in the tutorial.

Important

Using a project's endpoint requires configuring Microsoft Entra ID in your application. If you don't have Entra ID configured, use the Azure Application Insights connection string as indicated in step 3 of the tutorial.

Instrument the OpenAI SDK

When developing with the OpenAI SDK, you can instrument your code so traces are sent to Azure AI Foundry. Follow these steps to instrument your code:

Install azure-ai-projects, azure-monitor-opentelemetry, and opentelemetry-instrumentation-openai-v2 in your environment. The following example uses pip:
```
pip install azure-ai-projects azure-monitor-opentelemetry opentelemetry-instrumentation-openai-v2
```

Instrument the OpenAI SDK by using OpenAIInstrumentor():

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()

Get the connection string to the Azure Application Insights resource associated with your project. The following line uses the Azure AI Project client, which requires the use of Microsoft Entra ID for authentication:
```
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project_client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint="https://<your-resource>.services.ai.azure.com/api/projects/<your-project>",
)

connection_string = project_client.telemetry.get_connection_string()
```
Tip

Connection strings to Azure Application Insights looks like InstrumentationKey=aaaa0a0a-bb1b-cc2c-dd3d-eeeee4e4e4e;.... You can also access the connection string used in your project from the section Tracing in Azure AI Foundry portal. In the top navigation bar, select Manage data source and copy the Connection string. Configure your connection string in an environment variable.

Configure OpenTelemetry to send traces to the Azure Application Insights:

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor(connection_string=connection_string)

By default, OpenTelemetry doesn't capture inputs and outputs. Use the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true to capture them. Ensure this environment variable is configured in the environment level where your code is running.

Use the OpenAI SDK as usual:

client = project_client.get_azure_openai_client()

response = client.chat.completions.create(
    model="deepseek-v3-0324",
    messages=[
        {"role": "user", "content": "Write a short poem on open telemetry."},
    ],
)

If you go back to Azure AI Foundry portal, you should see the trace displayed:
It may be useful to capture sections of your code that mixes business logic with models when developing complex applications. OpenTelemetry uses the concept of spans to capture sections you're interested in. To start generating your own spans, get an instance of the current tracer object.
```
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
```

Then, use decorators in your method to capture specific scenarios in your code that you are interested in. Such decorators generate spans automatically. The following code example instruments a method called assess_claims_with_context with iterates over a list of claims and verify if the claim is supported by the context using an LLM. All the calls made in this method are captured within the same span:

def build_prompt_with_context(claim: str, context: str) -> str:
    return [{'role': 'system', 'content': "I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."},
            {'role': 'user', 'content': f"""
                The evidence is the following: {context}

                Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text.

                Claim:
                {claim}

                Assessment:
            """}]

@tracer.start_as_current_span("assess_claims_with_context")
def assess_claims_with_context(claims, contexts):
    responses = []
    for claim, context in zip(claims, contexts):
        response = client.chat.completions.create(
            model="gpt-4.5-preview",
            messages=build_prompt_with_context(claim=claim, context=context),
        )
        responses.append(response.choices[0].message.content.strip('., '))

    return responses

Traces look as follows:

You may also want to add extra information to the current span. OpenTelemetry uses the concept of attributes for that. Use the trace object to access them and include extra information. See how the assess_claims_with_context method has been modified to include an attribute:

@tracer.start_as_current_span("assess_claims_with_context")
def assess_claims_with_context(claims, contexts):
    responses = []
    current_span = trace.get_current_span()

    current_span.set_attribute("operation.claims_count", len(claims))

    for claim, context in zip(claims, contexts):
        response = client.chat.completions.create(
            model="gpt-4.5-preview",
            messages=build_prompt_with_context(claim=claim, context=context),
        )
        responses.append(response.choices[0].message.content.strip('., '))

    return responses

Trace to console

It may be useful to also trace your application and send the traces to the local execution console. Such approach may result beneficial when running unit tests or integration tests in your application using an automated CI/CD pipeline. Traces can be sent to the console and captured by your CI/CD tool to further analysis.

Configure tracing as follows:

Instrument the OpenAI SDK as usual:

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()

Configure OpenTelemetry to send traces to the console:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

span_exporter = ConsoleSpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
trace.set_tracer_provider(tracer_provider)

Use OpenAI SDK as usual:

response = client.chat.completions.create(
    model="deepseek-v3-0324",
    messages=[
        {"role": "user", "content": "Write a short poem on open telemetry."},
    ],
)

{
    "name": "chat deepseek-v3-0324",
    "context": {
        "trace_id": "0xaaaa0a0abb1bcc2cdd3d",
        "span_id": "0xaaaa0a0abb1bcc2cdd3d",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CLIENT",
    "parent_id": null,
    "start_time": "2025-06-13T00:02:04.271337Z",
    "end_time": "2025-06-13T00:02:06.537220Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "gen_ai.operation.name": "chat",
        "gen_ai.system": "openai",
        "gen_ai.request.model": "deepseek-v3-0324",
        "server.address": "my-project.services.ai.azure.com",
        "gen_ai.response.model": "DeepSeek-V3-0324",
        "gen_ai.response.finish_reasons": [
            "stop"
        ],
        "gen_ai.response.id": "aaaa0a0abb1bcc2cdd3d",
        "gen_ai.usage.input_tokens": 14,
        "gen_ai.usage.output_tokens": 91
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.31.1",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

Next steps

Trace agents using Azure AI Foundry SDK

Share via