CodePark/ questions / Logging, metrics & tracing
START PRACTICINGPRACTICE
Backend question · System design & architecture › Logging, metrics & tracing

A request passes through a gateway, two NestJS services and a Kafka topic. How do you make it possible to follow that one request end to end?

Give every request a trace context at the edge, and pass it along on every hop. The trace context is a small piece of data that carries the trace id. If the client already sent one, accept it.

How to pass it along:

  • With OpenTelemetry, each service creates spans for its work. A span is one timed piece of work.
  • On outgoing HTTP calls, the context travels in the W3C traceparent header.
  • For Kafka, the producer puts the context in message headers. The consumer continues the trace from them. This keeps the asynchronous hop linked.

Use the OpenTelemetry SDK's auto-instrumentation for HTTP, database and messaging clients. Auto-instrumentation creates spans for you without code changes. Add manual spans for important business steps. Send the data to a tracing backend through the OpenTelemetry Collector.

Also put the trace id in every structured log line. Then you can jump from a trace to its logs.

At high volume, use sampling, which means keeping only some traces:

  • Head sampling decides at the start of the request. It is cheap.
  • Tail sampling decides at the end. It keeps the interesting traces, such as errors and slow requests.
  • Create the trace context at the edge, or accept the one the client sent.
  • Pass the context in traceparent headers for HTTP and in message headers for Kafka.
  • Use OpenTelemetry auto-instrumentation plus manual spans for business steps.
  • Put the trace id in every structured log line.
  • Use sampling at high volume: head sampling is cheap, tail sampling keeps errors and slow requests.