Skip to content

Tracing

Status: Beta

Experimental: users report that this component is working, but it needs a broader audience to be battle-tested properly.

Please report any issues to the #nais channel on Slack.

New Grafana URL

When visualizing traces, you must use the new Grafana instance at https://grafana.nav.cloud.nais.io.

Traces are a record of the path a request takes through your application. They are useful for understanding how a request is processed in your application.

NAIS does not collect trace data automatically. If you want tracing integration, you must first instrument your application to collect traces, and then configure the tracing library to send it to the correct place.

Traces from NAIS applications are collected using the OpenTelemetry standard. Performance metrics are stored and queried from the Tempo component.

This documentation applies to backend applications. If you need to enable tracing for end users in their browser, see frontend observability.

Enabling trace collection

The first step in collecting trace information is to enable it in your application spec:

spec:
  observability:
    tracing:
      enabled: true

Next, you must select the appropriate OTLP exporter for your specific application. Ready-made libraries can be found for Java, Rust, Python, Go, and most other popular languages. Detailed instructions on how to configure specific tracing libraries are out of scope for this documentation.

Finally, the OTLP exporter must be configured to send data to the NAIS collector. That configuration is provided by NAIS through the $OTEL_EXPORTER_OTLP_ENDPOINT environment variable, which in turn is supposed to be automatically detected and used by your OpenTelemetry library. All data must be sent using the gRPC protocol.

Environment variables

The following environment variables are available for configuring the tracing library:

Name Description Default value
OTEL_SERVICE_NAME The name of the service. $APP_NAME
OTEL_RESOURCE_ATTRIBUTES A comma-separated list of key-value pairs to be added to the resource attributes. service.name=$APP_NAME,service.namespace=$APP_NAMESPACE
OTEL_EXPORTER_OTLP_ENDPOINT The endpoint to send trace data to. http://tempo-distributor.nais-system:4317
OTEL_EXPORTER_PROTOCOL The protocol to use when sending trace data. Must be grpc. grpc

Instrumenting your application

The following libraries are available for instrumenting your application:

Language Library
Java OpenTelemetry Java
Node.js OpenTelemetry Node.js
Python OpenTelemetry Python
Go OpenTelemetry Go

You can also see the nais/examples repository for examples of how to instrument your application.

Visualizing application performance

Visualization of traces can be done in the new Grafana installation.

You can use the Explore feature of Grafana with the prod-gcp-tempo and dev-gcp-tempo data sources.

There are no ready-made dashboards at this point, but feel free to make one yourself and contribute to this page.


Last update: 2023-11-06
Created: 2020-10-27