Monitoring .NET App Traces via OpenTelemetry & Zipkin Exporter

Requirements: Docker, .NET 7.0

While developing applications, monitoring them is also essential. To monitor metrics and traces, there are lots of open source solutions. And those solutions have their way of implementation to the application code.

With .NET 7.0, OpenTelemetry implementation got easier. Thanks to that, we can configure OpenTelemetry at bootstrap section and start collecting .NET metrics and application traces. But collecting traces is not enough. We also need to export those metrics and study them. So we need an exporter. OpenTelemetry has already implemented some of popular exporters as extensions. Selecting and configuring those exporters are up to you.

When a user says “The app is working slower”, you probably could not reply as “It works on my computer.” That’s why collecting traces to investigate them later is important. What services did that user’s request end up to? Which SQL queries has been initiated? How many milliseconds did pass? etc..

Open source tracing softwares are useful in those situations especially in distributed systems like microservices in cloud environments. Let’s see how we can add OpenTelemetry to our project. You can find the GitHub link to this project at the end of this article.

First of all, we create new .NET project. Let’s say the name of the project is “OpenTelemetryExample”

mkdir OpenTelemetryExample
cd OpenTelemetryExample
dotnet new webapi -o ./OpenTelemetryExample.API

You can open your project in your favorite IDE. I personally prefer VS Code. So my instructions will be according to the VS Code.

Program.cs is the file to bootstrap our .NET application. We do our configurations in this class and those configurations would live throughout the app. Like services you are going to inject and their lifetimes, logging options etc..

We need to add some NuGet packages for OpenTelemetry. If you are using Visual Studio, you can get them from NuGet Package Manager by writing the name of the packages in the manager. I am using VS Code, so I’ll use command line.

cd OpenTelemetryExample.API
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore --prerelease
dotnet restore

After we install our packages, we can start to our configurations. Let’s start with the imports.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Exporter;

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddConsoleExporter());

To describe;
.ConfigureResource() section is where we name our service.
.WithTracing() section is where we add our extensions.
.AddAspNetCoreInstrumentation() is where tell the OpenTelemetry to use .NET 7.0’s newly implemented support for OpenTelemetry by using its own tracing classes in System.Diagnostic. Before that, we had to write codes to all of our project to determine what should OpenTelemetry log.
.AddConsoleExporter() is where we can see the output of the exposed metrics in our console.

While we are in the directory of our API, let’s start the application by running the following command:

dotnet run

The output should be similar to the one below. Where it says “Now listening on” is which port our application listens to.

info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5049
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/idylmz/Projects/OpenTelemetryExample/OpenTelemetryExample.API
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]

Let’s send a simple “GET” request to the application via our browser. While creating a .NET API app, the template should include an endpoint to interact. We can use that endpoint to test our app. Since my port number is 5049, I am typing the below address to my browser and hit enter.

http://localhost:5049/weatherforecast

After I send my first request, I can see the output of OpenTelemetry on my console thanks to my ConsoleExporter.

ctivity.TraceId:            aaa58fe52611059db2b55b38d63ea4ec
Activity.SpanId:             5987a9685d6fa084
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName:        WeatherForecast
Activity.Kind:               Server
Activity.StartTime:          2023-05-31T14:48:51.4780440Z
Activity.Duration:           00:00:00.1581850
Activity.Tags:
    net.host.name: localhost
    net.host.port: 5049
    http.method: GET
    http.scheme: http
    http.target: /weatherforecast
    http.url: http://localhost:5049/weatherforecast
    http.flavor: 1.1
    http.user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36
    http.route: WeatherForecast
    http.status_code: 200
Resource associated with Activity:
    service.name: OpenTelemetryExample
    service.instance.id: eb77edbd-0dc4-484f-8cc6-157f7a73a5b2

If you can managed to accomplish so far, congratulations. You have successfully implemented OpenTelemetry to your .NET application. But in production, you could not search for messages in the console. Even if you look, it would be hard to find the correct messages. That’s where Zipkin comes in. It has a simple UI for you to search anything in your traces.

Zipkin (Zipkin.io)

Zipkin is used for collecting distributed traces and visualize service interactions with each other. In other words, service dependencies. OpenTelemetry has an extension like ConsoleExporter for Zipkin to export my service data. By using the below command, we can add ZipkinExporter to our .NET project.

dotnet add package OpenTelemetry.Exporter.Zipkin
dotnet restore

We have added ZipkinExporter to our project. But Zipkin itself is not around yet. Since the subject of this topic is not about “How to install Zipkin”, I will use Docker to run a Zipkin server. This little container will be responsible to act as a Zipkin server, collect my exporter metrics and show me them in its UI. After you run Docker, run below command to download Zipkin image.

docker pull openzipkin/zipkin

After the download of the image is completed, run below command to run a Zipkin container. We should map our 9411 port to container’s 9411 port to accept incoming requests to Zipkin server.

docker run -d -p 9411:9411 openzipkin/zipkin

By typing http://localhost:9411 to my browser, I make sure Zipkin is up and running. If it is, you should see a similar screen to this:

Now it is time to send our metrics to Zipkin. If we return to the code where we configured OpenTelemetry, we will add another line to add Zipkin Exporter.

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddConsoleExporter());

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddConsoleExporter()
                        .AddZipkinExporter());

Now let’s start our .NET app once again.

dotnet run

I am using the same http://localhost:5049/weatherforecast link to send a request. Then check it on Zipkin UI. By pressing “Run Query” button on the page, I am getting all traces without a filter.

As seen on the screenshot above, OpenTelemetry is exporting my traces and Zipkin is successfully collecting my traces.

We didn’t make so much configuratin. We didn’t even tell the OpenTelemetry where the Zipkin is running. It is because we are using everything in their default configurations. For example, when we don’t say where Zipkin is and only say .AddZipkinExporter(), the default configuration for that is, Zipkin server is running on localhost and on port 9411.

Let’s take a look at the configuration below. Now I am configuring Zipkin Exporter to send its data to a remote server. The configuration I am doing here is pointing to a virtual server in my private network. After you write “protocol://endpoint:port” information, you should add “/api/v2/spans” as below. Because this is where Zipkin greets its exporter metrics.

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddConsoleExporter()
                        .AddZipkinExporter(services =>
                        {
                            services.Endpoint = new Uri("http://10.0.0.5:9411/api/v2/spans");
                        }));

Bonus: SQL Traces

OpenTelemetry can also collect DB traces and send them to Zipkin server. That’s how you can monitor the situation of every SQL call in every trace. To do that, you should also add the proper NuGet package.

dotnet add package OpenTelemetry.Contrib.Instrumentation.EntityFrameworkCore --version 1.0.0-beta.6
dotnet restore

After that you should add another line to the configuration part of the OpenTelemetry in Program.cs. The last look of your code should be similar to the one below after you add “.AddEntityFrameworkCoreInstrumentation()”

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddEntityFrameworkCoreInstrumentation()
                        .AddConsoleExporter()
                        .AddZipkinExporter(services =>
                        {
                            services.Endpoint = new Uri("http://10.0.0.5:9411/api/v2/spans");
                        }));

We should also add and call our parameters from appsettings.json file which is a default configuration file of .NET Core web applications. Instead of writing hard coded strings to the compiled code, I should add those strings to the configuration file. If my parameters change, I should not constantly compile and deploy new code to production environment.

The default appsettings.json file looks like this:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
}

After we add some settings about OpenTelemetry, it should look like this.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "OpenTelemetrySettings":{
    "SQLTracingSettings":{
      "SetDbStatementForText": true,
      "SetDbStatementForStoredProcedure": true
    },
    "ZipkinSettings":{
      "Endpoint":"http://localhost:9411/api/v2/spans"
    }
  }
}

I am going to use OpenTelemetrySettings section to configure my Zipkin endpoint and what should OpenTelemetry collects. For instance, by default, OpenTelemetry only collects stored procedure calls. If we are not using stored procedures but raw queries or LINQ queries, we should set true “SetDbStatementForText” option. For more information, you can inspect its own GitHub page

Let’s edit our code one more time like below after we added our parameters to appsettings.json

builder.Services.AddOpenTelemetry()
                    .ConfigureResource(otelBuilder => otelBuilder
                        .AddService(serviceName: "OpenTelemetryExample"))
                    .WithTracing(otelBuilder => otelBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddEntityFrameworkCoreInstrumentation()
                        .AddConsoleExporter()
                        .AddZipkinExporter()
                        .ConfigureServices(services =>
                        {
                            services.Configure<ZipkinExporterOptions>(builder.Configuration.GetSection("OpenTelemetrySettings:ZipkinSettings"));
                            services.Configure<EntityFrameworkInstrumentationOptions>(builder.Configuration.GetSection("OpenTelemetrySettings:SQLTracingSettings"));
                        }));

Right now, it is reading parameters from appsettings.json. The keys we wrote are mapped into ZipkinExporterOptions and EntityFrameworkInstrumentationOptions classes. We can check if it works on Zipkin UI while our app is up and accepts requests.

And SQL queries would look like thhis. As you can see that trace is about 417 ms. There are 2 queries which lasted about 140 ms and 4 ms. While I successfully monitor my application, when someone complains about slow application, I can see what is wrong by just searching traces.