Skip to content

Repository files navigation

OpenTelemetry with .NET

The tiniest local-learning example I could think of.

Run

docker compose up
dotnet run --project Tel.Web

Grafana will be available (after a short while) on localhost:40300. Use admin as username and password. You can use JetBrains HTTP Client to simulate traffic by running the call defined in Tel.Web.http.

Features

  • Microsoft.Extensions.Logging based logging without clutter on stdout
  • Application startup logging. Before initializing OpenTelemetry
  • OTEL_DIAGNOSTICS.json file included
  • ASP .NET Instrumentation plus custom explicit tracing
  • Grafana LGTM included as observability platform
  • OpenTelemetry Collector Contrib. with configuration and self-monitoring included
  • ASP .NET minimal APIs
  • Source code commentary with links to reading resources
  • TODO Exemplars
  • TODO Samplers

Pictures

Human readable logs on stdout

Human Readable Logs on stdout

Application traces

Application traces

Application logs

Application logs

Application metrics

Application metrics

OpenTelemetry Collector metrics

Application metrics

Side notes

Pitfalls

This example uses OTLP over plain HTTP/protobuf because it keeps local traffic easy to inspect. Both http/protobuf and grpc are valid OTLP transports; production choices should be based on TLS, authentication, backend support, and deployment topology. See the official OTLP exporter configuration documentation.

Standard ASP.NET Core HTTP metrics are intentionally not registered in this example to de-noise the output and keep attention on one custom metric. See the official OpenTelemetry .NET metrics documentation for normal metrics instrumentation.

The explicit Collector pipelines intentionally omit batch and memory_limiter processors to make local signal visibility fast. These processors are normal production hardening components. See the official Collector processor documentation.

There are two OpenTelemetry Collectors: one baked in the Grafana LGTM container and one configured explicitly. This setup is for demonstration only. In production, use one deliberate Collector topology. See the official Collector configuration documentation.

The debug exporter and DEBUG Collector logs are useful for learning, but they are noisy and should not be copied as production defaults. See the official Collector exporter documentation.

The LGTM container has no persistence volume and this compose file has no health checks. That keeps the demo small, but telemetry data is ephemeral and readiness is verified manually. See the official Grafana Docker OpenTelemetry LGTM documentation.

Seemingly double metrics

You will see a job named opentelemetry-collector in Prometheus metrics. This exists because:

By default, the Collector generates basic metrics about itself and exposes them using the OpenTelemetry Go Prometheus exporter for scraping at http://127.0.0.1:8888/metrics. See here for mor information on default metrics

The Grafana LGTM distribution used in this example sets up scraping on localhost:8888 by default. See here the Grafana LGTM OpenTelemetry Collector configuration. The metrics for the OpenTelemetry Collector used within this example are under job name otelcol-contrib.

OpenTelemetry Configuration Schema

To better understand how OpenTelemetry service stanza can be configured browse the Go source code type definitions here.

OTEL_DIAGNOSTICS.json

Configures self-diagnostic for OpenTelemetry .NET SDK, the file is supposed to be full of NUL because it is pre-allocated. See here for further details

How do I change the labels that Loki assigns to my OTLP logs ?

See the official Loki documentation

Do I have the same concept for labels in Tempo, like Loki ?

No.

About

The tiniest example I could think of. Good as template.

Resources

Stars

1 star

Watchers

1 watching

Forks

Used by

Contributors

Languages