OpenTelemetry Tracing

Hive Router supports distributed tracing so you can follow requests across the gateway and your subgraphs.

This guide explains how to configure tracing in a practical, developer-friendly way: where to send traces, how to configure OTLP, how to tune throughput, and how to debug missing traces.

Choose your tracing destination

Hive Router supports three tracing paths. You can send traces directly to Hive Console through telemetry.hive.tracing, send them to an OTLP-compatible backend through telemetry.tracing.exporters, or use native Datadog tracing through a Datadog Agent.

Teams already running OpenTelemetry infrastructure usually prefer OTLP because it fits into existing collector pipelines and backend routing rules. Use native Datadog tracing when you need Datadog’s sampling, rate limiting, remote configuration, and all-request APM statistics.

Send traces to Hive Console

If you are already using Hive, sending traces to Console is usually the smoothest starting point. It keeps tracing data close to schema and usage insights, so it is easier to move from “this request is slow” to “which operation and field caused it”.

To make this work, Hive Router needs two pieces of information: an access token with permission to send traces, and a target reference. The target can be either a human-readable slug ($organizationSlug/$projectSlug/$targetSlug) or a target UUID (a0f4c605-6541-4350-8cfe-b31f21a4bf80).

With those values available as environment variables (HIVE_TARGET and HIVE_ACCESS_TOKEN), enable Hive tracing in the config file:

router.config.yaml

telemetry:
  hive:
    tracing:
      enabled: true
      # Optional for self-hosted Hive:
      # endpoint: https://api.graphql-hive.com/otel/v1/traces

After enabling tracing, send a few GraphQL queries through your router and open that same target’s Traces view in Hive Console. You should start seeing new traces for recent requests.

If traces do not appear, it usually means one of four things: tracing is not enabled, the token does not have necessary permissions, the configured target reference points to a different target, or the self-hosted endpoint is not reachable from the router runtime.

Send traces to OTLP-compatible backends

If your observability platform already supports OTLP ingestion, Hive Router can push traces straight to that OTLP endpoint. The destination can be an OpenTelemetry Collector or any system that natively understands OTLP.

router.config.yaml

telemetry:
  tracing:
    exporters:
      - kind: otlp
        enabled: true
        protocol: http
        endpoint: https://otel-collector.example.com/v1/traces
        http:
          headers:
            authorization:
              expression: |
                "Bearer " + env("OTLP_TOKEN")

Once configured, send normal requests through the router and check your backend for fresh traces.

router.config.yaml

telemetry:
  tracing:
    exporters:
      - kind: otlp
        enabled: true
        protocol: grpc
        endpoint: https://otel-collector.example.com:4317
        grpc:
          metadata:
            x-api-key:
              expression: env("OTEL_API_KEY")
          tls:
            # Optional SNI/verification override
            domain_name: otel-collector.example.com
            # Optional custom CA bundle
            ca: /etc/certs/ca.pem
            # Optional client cert for mTLS
            cert: /etc/certs/client.pem
            # Optional client key for mTLS
            key: /etc/certs/client.key

If gRPC export fails, metadata credentials and TLS files are usually the first places to inspect.

Send traces through native Datadog tracing

Native Datadog tracing uses the Datadog Rust tracer and sends traces to a Datadog Agent trace endpoint. This differs from sending generic OpenTelemetry data to a Datadog OTLP endpoint. It preserves Datadog’s native sampling behavior and computes request, error, and latency statistics from every recorded request, including requests whose detailed traces are not retained.

router.config.yaml

telemetry:
  tracing:
    collect:
      # Retain about 1% of detailed traces while keeping all-request APM statistics
      sampling: 0.01
    propagation:
      trace_context: true
    exporters:
      - kind: datadog
        enabled: true
        endpoint: http://datadog-agent:8126

enabled defaults to true. The endpoint is optional and can be a StringOrExpression. When omitted, Datadog uses its native configuration, including DD_TRACE_AGENT_URL, DD_AGENT_HOST, and DD_TRACE_AGENT_PORT. The Agent trace endpoint normally uses port 8126; it is not the OTLP gRPC endpoint commonly exposed on port 4317.

The router resolves and validates a configured endpoint at startup. An unresolved expression, malformed URL, unsupported Agent URL scheme, or more than one enabled Datadog exporter prevents startup. The router does not probe Agent connectivity. An unavailable Agent therefore does not stop the router from starting, and asynchronous export failures are reported through Datadog logging.

Sampling and mixed exporters

Hive Router uses one shared tracer provider. When a Datadog exporter is enabled, Datadog owns the provider-wide, parent-aware sampling decision. The router maps collect.sampling to Datadog’s catch-all sample rate, overriding DD_TRACE_SAMPLE_RATE. More specific DD_TRACE_SAMPLING_RULES can apply before that rate, and Datadog remote configuration can update native sampling behavior.

parent_based_sampler controls whether Datadog inherits the sampling decision of an incoming remote parent:

parent_based_sampler: true - a sampled remote parent (for example, a traceparent ending in -01) stays sampled, whatever collect.sampling says.
parent_based_sampler: false (default) - the router marks the inherited decision as deferred, so Datadog evaluates the trace with its own sampling policy: collect.sampling, Datadog sampling rules, and rate limits.

In both cases the incoming trace ID, parent span ID, and trace state are preserved, so the trace stays connected to its parent. For example, with the configuration below, a request with a sampled traceparent does not have its detailed trace retained, but Datadog still receives its request and error statistics:

router.config.yaml

telemetry:
  tracing:
    collect:
      sampling: 0.0
      parent_based_sampler: false
    exporters:
      - kind: datadog

When explicit sampling is active, Datadog retains at most 100 detailed traces per second by default. Set DD_TRACE_RATE_LIMIT to change this ceiling. At high request rates, the retained percentage can be lower than collect.sampling, while all-request request, error, and latency statistics remain complete. For the same reason, sampling: 0.0 keeps instrumentation active when Datadog is enabled: no detailed traces are retained, but Datadog can still compute its APM statistics.

You can attach OTLP, stdout, and Hive trace processors to the same Datadog-backed provider. Those processors receive only sampled spans and ignore Datadog’s RecordOnly spans. Because sampling is provider-wide, independent per-exporter sampling rates are not supported.

DD_TRACE_ENABLED=false disables the Datadog tracer and every OTLP, stdout, and Hive processor on the shared Datadog-backed provider. To disable only Datadog, disable its router exporter instead:

router.config.yaml

telemetry:
  tracing:
    exporters:
      - kind: datadog
        enabled: false
      - kind: otlp
        enabled: true
        protocol: grpc
        endpoint: https://otel-collector.example.com:4317

Without an enabled Datadog exporter, the router uses its generic OpenTelemetry provider and sampler. Router resource attributes are passed to Datadog, where native Datadog precedence applies to the equivalent DD_SERVICE, DD_ENV, and DD_VERSION values.

The router continues to use the propagation formats configured under telemetry.tracing.propagation, including W3C Trace Context. It does not install Datadog-native propagation. The Datadog-backed provider also uses the router’s existing force-flush and shutdown lifecycle.

For native Datadog behavior and configuration details, see the datadog-opentelemetry API, Datadog Rust tracing configuration, trace metrics, and ingestion mechanisms.

Production baseline

For production workloads, define a clear service identity, begin with conservative sampling rates, and use a single primary propagation format.

router.config.yaml

telemetry:
  resource:
    attributes:
      service.name: hive-router
      service.namespace: your-platform
      deployment.environment:
        expression: env("ENVIRONMENT")
  tracing:
    collect:
      # Trace about 10% of requests
      sampling: 0.1
      # Respect upstream sampling decisions
      parent_based_sampler: true
    propagation:
      # Recommended default
      trace_context: true
      baggage: false
      b3: false
      jaeger: false
    exporters:
      - kind: otlp
        enabled: true
        protocol: grpc
        endpoint: https://otel-collector.example.com:4317

This configuration is designed to be a safe, predictable starting point. It gives each deployment a clear identity in your telemetry backend, keeps trace volume under control, and sticks to a single propagation format.

In practice, this means you’ll see enough traces to understand real production behavior without overwhelming storage or blowing up costs.

Batching and throughput tuning

Batching settings control how traces move from the router to your OTLP endpoint. You’re able to tune these settings to control delivery latency of traces, resilience during traffic spikes and memory pressure on the router.

Field	You’d usually increase this when	Tradeoff
`max_queue_size`	Traces are dropped during traffic spikes	Higher memory usage
`max_export_batch_size`	You want better export throughput per flush	Potentially higher burst latency
`scheduled_delay`	You want fewer export calls (`higher`) or lower latency (`lower`)	Throughput vs latency
`max_export_timeout`	Your OTLP endpoint or network is occasionally slow	Longer waits on blocked exports
`max_concurrent_exports`	Your OTLP endpoint can handle more parallel uploads	Higher downstream pressure

As a quick rule:

if traces arrive late, lower scheduled_delay.
if traces drop under burst load, increase max_queue_size first.
if your OTLP collector has headroom, raise max_concurrent_exports.

Propagation

Propagation settings control how trace context flows between clients, the router, and subgraphs. In most modern OpenTelemetry setups, trace_context is the safest default.

You should only enable b3 or jaeger when those formats are required by other components.

If clients send custom tracing headers, make sure your CORS configuration allows those headers through.

Compliance with OpenTelemetry Semantic Conventions

OpenTelemetry has standardized attribute names used on spans. Those conventions ensure that telemetry produced by different services, libraries, and vendors is consistent and understandable across tools.

The behavior is controlled by telemetry.tracing.instrumentation.spans.mode, which selects which attribute set is written to spans:

spec_compliant (default) - emits only the stable attributes
deprecated - emits only the deprecated attributes
spec_and_deprecated - emits both stable and deprecated attributes

router.config.yaml

telemetry:
  tracing:
    instrumentation:
      spans:
        mode: spec_compliant

Most teams should stay on spec_compliant. The other modes are primarily useful when migrating legacy dashboards that still expect deprecated attributes.

Span status and GraphQL errors

GraphQL errors are usually returned in an HTTP 200 response. The router still marks these requests as failed, so tracing backends such as Datadog count them in operation-level and service-level error rates:

The graphql.operation span gets the Error status when the response contains at least one GraphQL error.
The root http.server span gets the Error status when the GraphQL response contains errors, even if the HTTP status is 200.
For HTTP 5xx responses, the root span gets the Error status and error.type is set to the bare status code as a string, such as "500".
Otherwise, the status of 1xx, 2xx, and 3xx responses is left unset.

A response can contain several errors with different codes, so the router does not pick one of them as an aggregate error.type for the graphql.operation span or the root span. The root span’s error.type is set only for HTTP 5xx responses, as described above. The individual errors are recorded on the graphql.operation span instead:

hive.graphql.error.count - the number of errors in the response.
hive.graphql.error.codes - the distinct error codes, sorted and comma-separated, such as NOT_FOUND,UNAUTHENTICATED. Omitted when no error carries a code.
One span event per error. The event’s name is the error message, and the event is tagged with hive.kind: graphql.error, so filter on that attribute rather than on the event name. Each event also carries error.message, error.type (the error’s extensions.code, or unknown), and, when known, hive.error.subgraph_name, hive.error.path, and hive.error.affected_path.

Subgraph errors are recorded the same way on the graphql.subgraph.operation span of the subgraph request that produced them.

Subscription errors

GraphQL errors produced while establishing or running WebSocket subscriptions are recorded in traces like errors from HTTP GraphQL requests. This includes authentication, parsing, validation, and execution errors, making subscription failures visible in the same tracing backend as other router traffic.

Troubleshooting

When traces are missing or incomplete, think in layers:

exporter setup
sampling behavior
propagation
transport

If no traces appear at all, verify if the exporter is enabled, the endpoint is reachable, and credentials are valid.

If spans show up but links are broken, propagation formats are usually misaligned between services.

If under high load, traces are delayed or dropped, then often it’s a batch processor issue. In that case tune the batch processor settings and observe.

Configuration reference

For all options and defaults, see telemetry configuration reference.