OpenTelemetry Tracing

Trace every request with OpenTelemetry: one SERVER span per request, named by route pattern, continuing the caller's trace from a W3C traceparent header. Send the spans to Jaeger, Grafana Tempo, Honeycomb, Datadog or any OTLP collector. Requires the otel Cargo feature.

ultimo = { version = "0.9", features = ["otel"] }

Ultimo depends only on the opentelemetry API crate. Your app picks the SDK, sampler and exporter and installs them globally; until it does, every span is a no-op and costs next to nothing.

Quick start (OTLP)

[dependencies]
ultimo = { version = "0.9", features = ["otel"] }
opentelemetry = "0.33"
opentelemetry_sdk = "0.33"
opentelemetry-otlp = "0.33"   # HTTP/protobuf exporter, port 4318
tokio = { version = "1", features = ["full"] }

use opentelemetry::global;
use opentelemetry_sdk::propagation::TraceContextPropagator;
use opentelemetry_sdk::trace::SdkTracerProvider;
use opentelemetry_sdk::Resource;
use ultimo::otel::Tracing;
use ultimo::prelude::*;
 
#[tokio::main]
async fn main() -> ultimo::Result<()> {
    // 1. Install an SDK and the W3C propagator once, at startup.
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .build()
        .expect("OTLP exporter");
    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_resource(Resource::builder().with_service_name("my-api").build())
        .build();
    global::set_tracer_provider(provider.clone());
    global::set_text_map_propagator(TraceContextPropagator::new());
 
    // 2. Register the middleware first so the span covers the whole chain.
    let mut app = Ultimo::new();
    app.use_middleware(Tracing::new().middleware());
 
    app.get("/users/:id", |ctx: Context| async move {
        ctx.json(json!({ "id": ctx.req.param("id")? })).await
    });
 
    let result = app.listen_with_shutdown("0.0.0.0:3000", ultimo::shutdown_signal()).await;
    // 3. Flush buffered spans before exiting.
    let _ = provider.shutdown();
    result
}

The exporter reads the standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS environment variables, so the same binary can point at a local collector or a hosted backend.

What's recorded

Spans follow the HTTP semantic conventions:

• 5xx responses set the span status to Error. 4xx don't: on a server span they're the client's mistake, not a server failure.
• Timed-out or abandoned requests (see request_timeout) still end their span, marked error.type = "cancelled".
• The query string isn't recorded — it often carries tokens or personal data.
• Requests that match no route never reach middleware, so 404s for unknown paths produce no span.

Distributed traces

With TraceContextPropagator installed, an incoming traceparent (and tracestate) header makes the server span a child of the caller's span, so a browser, gateway or upstream service and your Ultimo API show up in one trace. Without a propagator installed, every request starts a new trace.

Child spans in handlers

While the rest of the middleware chain and your handler run, the server span is the current OpenTelemetry context. Spans you start there become its children:

use opentelemetry::global;
use opentelemetry::trace::{Span, Tracer};
 
app.get("/users/:id", |ctx: Context| async move {
    let mut span = global::tracer("my-api").start("db.load_user");
    let user = load_user(ctx.req.param("id")?).await?;
    span.end();
    ctx.json(user).await
});

Need the trace id, say for an error page or a log line?

use opentelemetry::trace::TraceContextExt;
 
let trace_id = opentelemetry::Context::current()
    .span()
    .span_context()
    .trace_id()
    .to_string();

Work you tokio::spawn doesn't inherit the context automatically. Capture opentelemetry::Context::current() and wrap the task with opentelemetry::context::FutureExt::with_context.

Correlating with logs and request ids

Register request_id() after the tracing middleware and each span carries ultimo.request_id, the same value returned in the X-Request-Id response header and by ctx.request_id(). To put trace ids on your tracing log lines, add tracing-opentelemetry to your subscriber.

Custom scope name

Spans are reported under the instrumentation scope ultimo. To use your own:

app.use_middleware(Tracing::with_scope_name("my-api").middleware());

Metrics

Tracing pairs with the metrics feature, which serves request counts and latency histograms to Prometheus. Use both: metrics to see that something is slow, traces to see why.

Example

examples/otel-tracing is a browser page that sends requests with a fresh traceparent and lists the resulting spans (server span plus a child span from the handler), using an in-memory exporter so it runs without a collector:

cargo run -p otel-tracing-example

See also: Metrics · Middleware · Deployment · API Reference.