OpenTelemetry is the standard way to get telemetry out of an application, and it is now the default answer whenever someone asks “how do we see what our services are actually doing?”. This guide teaches OpenTelemetry from zero: what it is, the vocabulary, and how traces, metrics and logs really work, with runnable Python examples and a local Docker stack you can point a real application at.

Every command and every block of output in this guide was run against OpenTelemetry Python SDK 1.44.0 and Jaeger 1.57. Nothing here is illustrative-only.

What is OpenTelemetry?

OpenTelemetry (usually shortened to OTel) is an open standard, plus a set of libraries, for producing telemetry — the data a running program emits about what it is doing.

It defines three things:

• What the data looks like. Traces, metrics and logs share one data model and one set of attribute names.
• How programs produce it. A stable API you call from code, and an SDK that does the recording.
• How it travels. OTLP, one wire protocol that every modern observability backend accepts.

Here is the part people get wrong most often: OpenTelemetry does not store or display anything. There is no OpenTelemetry dashboard, no OpenTelemetry database, no OpenTelemetry alerting. Storage, search, dashboards and alerts all come from a backend — SigNoz, Jaeger, Grafana Tempo, Honeycomb, Datadog and so on.

That split is the entire point:

YOUR CODE  ──(OTel API: standard)──►  OTel SDK  ──(OTLP: standard)──►  ANY BACKEND
   write once                        config only                       swappable

Before OpenTelemetry, adding tracing meant installing a vendor’s agent and writing vendor-specific calls throughout your code. Switching vendors meant rewriting all of it. With OTel, the instrumentation in your code never changes — you point an environment variable somewhere else.

What is it actually for?

Three questions, three signals. Learn them as three different questions, because confusing them is the most expensive mistake in observability:

The real superpower is correlation. All three carry the same resource attributes, and log records carry the trace_id of the span that was active when they were written. A typical investigation runs: a metric alert tells you something is wrong, a trace shows you where, and the logs inside that trace tell you what. Without OTel those are three disconnected tools and three separate guesses.

The vocabulary, in one table

Read this once, skim it again after the examples. Everything below gets its own section later.

Demo 1: your first trace in 60 seconds

No Docker, no backend, no account. Just print spans to the terminal.

Install the SDK:

python3 -m venv .venv && source .venv/bin/activate
pip install opentelemetry-sdk

Save this as hello_trace.py:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

# 1. Configure the SDK once, at startup
resource = Resource.create({"service.name": "hello-otel"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

# 2. Get a tracer (one per module, at import time)
tracer = trace.get_tracer("demo.hello")

# 3. Create spans
with tracer.start_as_current_span("make.breakfast") as parent:
    parent.set_attribute("menu", "eggs")
    with tracer.start_as_current_span("boil.water") as child:
        child.set_attribute("litres", 1)

Run it:

python hello_trace.py

Real output, trimmed to the interesting fields:

{
    "name": "boil.water",
    "context": {
        "trace_id": "0x8223c45a90b137076e5728ad10956d26",
        "span_id": "0x57d1b866b9be98df"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xf36113f66f828bb1",
    "start_time": "2026-09-21T09:47:59.731073Z",
    "end_time": "2026-09-21T09:47:59.731087Z",
    "status": { "status_code": "UNSET" },
    "attributes": { "litres": 1 },
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.version": "1.44.0",
            "service.name": "hello-otel"
        }
    }
}
{
    "name": "make.breakfast",
    "context": {
        "trace_id": "0x8223c45a90b137076e5728ad10956d26",
        "span_id": "0xf36113f66f828bb1"
    },
    "parent_id": null,
    "attributes": { "menu": "eggs" }
}

Four things to notice, and they teach most of the model:

1. Both spans share one trace_id. They are one trace.
2. boil.water has parent_id = make.breakfast’s span_id. The with block nesting created the tree. You did not pass a parent anywhere — the SDK tracked it through the context.
3. make.breakfast has parent_id: null. That makes it the root span.
4. The child printed first. A span is only exported after it ends, and the child ended first. This surprises people debugging locally — export order is not tree order.

Also note "status_code": "UNSET". That is what a successful span looks like. UNSET is not an error or a missing value; only ERROR means failure.

Anatomy of a span

That JSON is the whole data model. Field by field:

Span kind

The kind states the span’s role in a conversation between components.

A CLIENT span in service A with a SERVER child in service B is the standard shape of one network hop.

Traces are trees

Follow the parent_id pointers and you get a tree. Backends draw it as a waterfall — each bar placed by start time, sized by duration:

trace 4bf92f35…            0ms        200        400        600        800
POST /graphql              [=================================================] 820 ms
└ resolve startLesson        [=============================================]   760 ms
  ├ mongo find learner       [===]                                              45 ms
  ├ GET assets CDN               [====================================]        610 ms
  └ render prompt                                                    [==]      30 ms

Two things to read off any waterfall:

• Critical path — the chain that decides total duration. Here it is the CDN fetch. Optimising the prompt render would change nothing, and this is the single most common way tracing saves wasted engineering effort.
• Self time — time a span spends not covered by its children. resolve startLesson = 760 − (45 + 610 + 30) = 75 ms of its own work. Large self time with no children usually means an uninstrumented call or real CPU work.

Instrumenting a function properly

This is the pattern worth memorising. One function, every exit path visible:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("demo.assets")

def download(url: str, path: Path) -> bool:
    parsed = urlparse(url)
    with tracer.start_as_current_span("asset.download") as span:
        span.set_attribute("url.domain", parsed.hostname or "")
        span.set_attribute("url.path", parsed.path)   # never the query string: signed URLs carry credentials
        written = 0
        try:
            resp = requests.get(url, stream=True, timeout=30)
            span.set_attribute("http.response.status_code", resp.status_code)
            resp.raise_for_status()
            with open(path, "wb") as f:
                for chunk in resp.iter_content(8192):
                    written += len(chunk)
                    f.write(chunk)
            span.set_attribute("asset.bytes_written", written)
            span.set_attribute("asset.download.outcome", "success")
            return True
        except Exception as exc:
            span.set_attribute("asset.bytes_written", written)
            span.set_attribute("asset.download.outcome", "error")
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            return False

Why this specific shape:

• One outcome attribute on every exit path. “Count by asset.download.outcome” now gives you a success/error rate with no joins and no second metric.
• The success path records what the failure path records. Without bytes_written on successes you cannot compare them against failures.
• The span opens outside the try, so the failure is inside the span rather than beside it.
• Whatever was known before the failure is kept. A timeout span has no http.response.status_code at all, and that absence is itself information.

The caught-exception trap

start_as_current_span records exceptions automatically — but only ones that escape the block. Code that catches an exception and returns False produces a span that looks perfectly successful:

# BROKEN: the span ends with status UNSET, as if nothing went wrong
with tracer.start_as_current_span("asset.download") as span:
    try:
        download()
    except Exception:
        return False        # ← the span never learns about this

Always record_exception() and set_status(ERROR) yourself when you swallow an exception.

Attributes, events or child spans?

Attributes accept str, bool, int, float, bytes, and homogeneous lists of those. Anything else — None included — is dropped with a warning. A span keeps at most 128 attributes, 128 events and 128 links by default; extras vanish, leaving only a dropped_attributes_count.

Demo 2: a real backend with Docker

The console exporter is fine for learning, useless for looking at a trace tree. Let us send real traces to Jaeger.

Create docker-compose.yaml:

services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    container_name: otel-demo-jaeger
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP

Start it:

docker compose up -d
curl -s -o /dev/null -w "jaeger UI: %{http_code}\n" http://localhost:16686/

jaeger UI: 200

Now an application that exports over OTLP and gets free spans from an instrumentation library:

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc \
            opentelemetry-instrumentation-requests requests

import requests
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode

resource = Resource.create({"service.name": "asset-fetcher", "deployment.environment": "demo"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)

RequestsInstrumentor().instrument()   # every requests call now gets a CLIENT span, free

tracer = trace.get_tracer("demo.assets")

def download(url: str) -> bool:
    with tracer.start_as_current_span("asset.download") as span:
        span.set_attribute("url.domain", "localhost")
        written = 0
        try:
            resp = requests.get(url, timeout=2)
            span.set_attribute("http.response.status_code", resp.status_code)
            resp.raise_for_status()
            written = len(resp.content)
            span.set_attribute("asset.bytes_written", written)
            span.set_attribute("asset.download.outcome", "success")
            return True
        except Exception as exc:
            span.set_attribute("asset.bytes_written", written)
            span.set_attribute("asset.download.outcome", "error")
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            return False

with tracer.start_as_current_span("batch.fetch_assets"):
    download("http://localhost:16686/")            # succeeds
    download("http://localhost:9999/unreachable")  # nothing is listening on 9999

provider.force_flush()
provider.shutdown()

Run it, then open http://localhost:16686 and pick the asset-fetcher service. The trace that arrives, read back from Jaeger’s API:

traceID 3af067192411bb1cc410b8d544d2a0c0
  batch.fetch_assets   dur=  20873us parent=ROOT   kind=internal outcome=-
  asset.download       dur=  10273us parent=6c91…  kind=internal outcome=success
  GET                  dur=   4852us parent=ebd4…  kind=client   outcome=-
  asset.download       dur=  10478us parent=6c91…  kind=internal outcome=error    ERROR
      event: exception
  GET                  dur=   7696us parent=da23…  kind=client   outcome=-        ERROR
      event: exception

The failing span’s tags and its exception event:

{
 "url.domain": "localhost",
 "asset.bytes_written": 0,
 "asset.download.outcome": "error",
 "span.kind": "internal",
 "error": true
}
EVENT: exception | requests.exceptions.ConnectionError |
       HTTPConnectionPool(host='localhost', port=9999): Max retries exceeded …

Everything the pattern promised is visible. Both outcomes are queryable by one attribute. The GET child spans appeared without a single line of tracing code, because RequestsInstrumentor patched the library. And the failed span has no http.response.status_code key at all — no response ever arrived, and the absence distinguishes a connection failure from an HTTP error.

For a fuller Jaeger walkthrough, see How to Setup and Run Jaeger With Docker and Docker Compose.

Instrumentation libraries are where the value is

Before writing manual spans, install the instrumentors. Each one patches a library to emit CLIENT/SERVER spans and handle context propagation:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor

FastAPIInstrumentor.instrument_app(app, excluded_urls="/health,/ready")
RequestsInstrumentor().instrument()
Psycopg2Instrumentor().instrument()

They are all named opentelemetry-instrumentation-<library>: requests, httpx, aiohttp-client, fastapi, flask, django, pymongo, sqlalchemy, psycopg2, redis, botocore, logging, and dozens more. There is also fully zero-code instrumentation:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install     # installs instrumentors for libraries it finds
OTEL_SERVICE_NAME=my-svc opentelemetry-instrument python app.py

A library without an enabled instrumentor emits nothing, and its calls show up as unexplained self time in the parent span.

Context and propagation

How did boil.water know its parent without being told? Context.

Context is an immutable bag of values that follows your code while it runs, stored in a Python contextvars.ContextVar. OTel keeps two things in it: the current span and the baggage. The analogy that sticks: a trace is a relay race, context is the baton, propagation is handing the baton to the next runner, and baggage is a note taped to the baton that every runner can read.

The gotcha that breaks real traces

Context follows some execution hops and not others. This table is worth pinning up:

When the parent is not visible, the new span has no parent, becomes the root of a brand-new trace, and silently loses all baggage. Here is that failure and its fix, proven side by side:

import contextvars
from concurrent.futures import ThreadPoolExecutor

def work(tag):
    with tracer.start_as_current_span(f"work.{tag}"):
        pass

with tracer.start_as_current_span("parent"):
    with ThreadPoolExecutor(max_workers=2) as ex:
        ex.submit(work, "broken").result()                            # context lost
        ex.submit(contextvars.copy_context().run, work, "fixed")      # context carried

Real output — span name, first 8 hex of the trace ID, and parent span ID:

span          trace (first 8)   parent
work.broken   c1d77371          NONE
work.fixed    eb7bb5b0          c97363f09d91f531
parent        eb7bb5b0          NONE

work.broken landed in a different trace with no parent. work.fixed shares the parent’s trace and points at it. If you have ever wondered why half your background work shows up as thousands of single-span traces, this is why.

Three other fixes work: use asyncio.to_thread instead of run_in_executor; capture context.get_current() and context.attach() it inside the worker; or install opentelemetry-instrumentation-threading, which patches thread creation globally.

Call copy_context() once per task. A contextvars.Context can be entered by only one thread at a time. Reusing one copy across concurrent submits raises RuntimeError: cannot enter context: … is already entered.

Crossing service boundaries

Between processes, the context travels in an HTTP header defined by the W3C Trace Context standard:

traceparent: 00-f541076822162e1af8f8d80cfbc64929-b776db45cabdf5a8-03
             ││ │                                │                │
             ││ │                                │                └─ trace-flags
             ││ │                                └─ parent-id: the SENDER's span_id
             ││ └─ trace-id (32 hex), identical across the whole trace
             │└─ separator
             └─ version (always 00 today)

Trace flags are a bit field: 01 = sampled (“I am recording this trace, so record your part too”), 02 = random trace ID, so 03 = both. Instrumented HTTP clients inject this header; instrumented servers extract it. That is the entire mechanism behind distributed tracing.

For queues, WebSockets or any non-HTTP transport, do it by hand:

from opentelemetry import propagate

# producer
carrier = {}
propagate.inject(carrier)              # {'traceparent': '00-…-01', 'baggage': '…'}
queue.send(body=payload, attributes=carrier)

# consumer
ctx = propagate.extract(message.attributes)
with tracer.start_as_current_span("queue.process", context=ctx, kind=SpanKind.CONSUMER):
    ...

Baggage

Baggage is string key/values you attach to the context so everything downstream can read them — including other services, via a baggage header.

from opentelemetry import baggage, context

ctx = baggage.set_baggage("client_session_id", "cs-123")
token = context.attach(ctx)

Two things to know. First, baggage is not telemetry by itself — it rides in the context but never lands on a span unless a processor such as BaggageSpanProcessor copies it across at span start. Second, the baggage header is added to every instrumented outgoing call, including to third parties. Never put secrets or personal data in it.

Metrics

Metrics answer aggregate questions cheaply. Where a trace is one row per operation, a metric is a handful of numbers per export interval, no matter how much traffic you serve.

Instruments

from opentelemetry import metrics

meter = metrics.get_meter("demo.assets")

downloads = meter.create_counter("asset.downloads", unit="1", description="Downloads by outcome")
duration = meter.create_histogram("asset.download.duration", unit="ms")

downloads.add(1, {"outcome": "success", "source": "s3"})
duration.record(87.0, {"source": "s3"})

Histograms do not keep your values

This trips everyone up eventually. A histogram keeps counts per bucket, plus count, sum, min and max. It does not store the individual measurements. Recording 87, 143, 92 and 2100 ms with bucket bounds [100, 250, 500, 1000, 2500] produces exactly this:

{'bounds': [100, 250, 500, 1000, 2500],
 'counts': [2, 1, 0, 0, 1, 0],
 'count': 4, 'sum': 2422, 'min': 87, 'max': 2100}

bucket         count
≤100 ms        ██ 2        ← 87, 92
100–250 ms     █ 1         ← 143
250–500 ms       0
500–1000 ms      0
1000–2500 ms   █ 1         ← 2100
>2500 ms         0

The backend estimates percentiles by interpolating inside a bucket. Percentiles from histograms are therefore approximate, and only as precise as the bucket layout.

The default buckets assume milliseconds. Without configuration the SDK uses [0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]. Record seconds into those and everything from 0 to 5 s lands in the first two buckets, making every percentile useless. Either record in ms, or set your own bounds.

Cardinality is the whole cost model

Every distinct combination of attribute values is a separate stored time series. The backend indexes and bills each one.

asset.downloads{outcome, source, region}
  outcome ∈ {success, error, abandoned} = 3
  source  ∈ {s3, http}                  = 2
  region  ∈ {us-west-2, eu-west-1}      = 2
  → 3 × 2 × 2 = 12 series                        fine

add client_session_id (20,000 sessions/day)
  → 12 × 20,000 = 240,000 series                 cost explosion, slow queries

Never put unbounded IDs on a metric. User IDs, session IDs, request IDs, URLs containing IDs and raw error messages all belong on spans or logs, where high cardinality is normal and valuable. Metric attributes should come from small fixed sets.

The escape hatch, when an instrumentation library emits attributes you cannot afford, is a View — an SDK rule that reshapes a metric before export:

from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation, DropAggregation

views = [
    View(instrument_name="asset.download.duration",
         aggregation=ExplicitBucketHistogramAggregation([100, 250, 500, 1000, 2500])),
    View(instrument_name="asset.downloads", attribute_keys={"outcome"}),   # keep ONLY outcome
    View(instrument_name="http.client.request.body.size", aggregation=DropAggregation()),
]

Four add() calls carrying client_session_id values s1, s2, s3, s1, through that middle View:

{'name': 'asset.downloads', 'attrs': {'outcome': 'success'}, 'value': 3}
{'name': 'asset.downloads', 'attrs': {'outcome': 'error'},   'value': 1}

The session IDs are gone, and two time series remain instead of four.

Temporality: cumulative or delta

OTLP defaults to cumulative; many hosted backends prefer delta because it is stateless for them. Set it with OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta.

There is a real trap here. Counters and histograms benefit from delta, but up-down counters should stay cumulative, because they measure a level, not events. The delta of a level is a net change and is as often negative as positive — an http.server.active_requests exported as delta can read a minimum of -2 over a week, which is meaningless, and no peak value is recoverable from it.

Metrics or spans?

Logs and correlation

OTel does not ask you to replace your logging library. It bridges it. A handler forwards records from Python’s logging into the OTel pipeline, where each one picks up the trace_id and span_id of whatever span was current:

import logging
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
set_logger_provider(logger_provider)
logging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))

logging.info("cache warmed")   # carries trace_id automatically if a span is active

That single trace_id on each record is what turns three tools into one investigation. You can also use LoggingInstrumentor to inject the IDs into your plain-text log lines, which is useful when the text logs go somewhere else entirely.

Logs are the most expensive signal per unit of insight. Prefer a span attribute over a log line where you can — it is structured, queryable, and already correlated. Where log volume is a concern, sample by severity: ship everything in staging, warnings and errors in production.

The SDK pipeline

Everything so far happens inside one pipeline, configured once at startup:

start span → Sampler → (drop | record) → processor.on_start() → your code
          → span.end() → processor.on_end() → batch queue → Exporter → backend

API versus SDK

This split is deliberate: libraries depend on the tiny API and can ship instrumentation without forcing an SDK on anyone. But it produces the single most common support question in observability.

No SDK means silence, not an error. If set_tracer_provider(...) is never called — because a gate flag is off, setup crashed, or this particular entrypoint never runs it — every span becomes a no-op and nothing complains. Debug it with print(type(trace.get_tracer_provider())). A ProxyTracerProvider means the SDK was never installed in this process.

Providers and resource

A provider (TracerProvider, MeterProvider, LoggerProvider) owns the resource, the processors, the sampler and the limits. The resource describes the emitting process and attaches to every span, metric and log it produces.

resource = Resource.create({
    "service.name": "agent-backend",          # the single most important attribute
    "service.version": "2026.13.52",
    "deployment.environment": "production",
    "k8s.pod.name": os.environ.get("HOSTNAME", ""),
})

Use Resource.create(), not Resource(). Only create() merges the SDK defaults plus OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME; the plain constructor sends exactly what you passed and silently ignores those environment variables.

Processors

BatchSpanProcessor defaults: export every 5 s or 512 spans, queue capacity 2048, export timeout 30 s. All tunable with OTEL_BSP_*.

A full queue drops spans silently. When spans arrive faster than the exporter drains them — a slow or unreachable backend, a traffic spike — the oldest queued span is discarded on each new arrival, with no log line. Sustained export failures look like “some spans are missing”, not like an error.

You can also write your own processor. Enrichment must happen in on_start, because an ended span is read-only:

class StampRegion(SpanProcessor):
    def on_start(self, span, parent_context=None):
        span.set_attribute("region", self._region)     # still writable here
    def on_end(self, span):
        pass                                            # ReadableSpan: read-only

on_start runs inline on the thread that started the span, inside your request. Keep it fast and never block in it.

Exporters and OTLP

Hosted backends usually accept OTLP on 443 with TLS plus an auth header. Configure it all with environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.example.com:443
OTEL_EXPORTER_OTLP_HEADERS=api-key=<from a secret, never committed>
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_COMPRESSION=gzip

Endpoint path rules cause most OTLP 404s. With the HTTP exporter, the generic OTEL_EXPORTER_OTLP_ENDPOINT gets /v1/traces appended automatically. A signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is used exactly as given. Mixing them up is the classic mistake.

Want to send to two backends at once? Add one batch processor per destination. They both see every sampled span:

provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="https://vendor-a:443")))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="https://vendor-b:443")))

This “dual-emit” pattern is how you migrate between vendors with zero risk: run both, rebuild your dashboards on the new one, then delete the old exporter.

Flushing on exit

Batching means recent spans sit in memory until the next export. Providers register an atexit hook that flushes on a normal interpreter exit — but atexit does not run on SIGKILL, os._exit(), a hard crash, or a Kubernetes pod killed past its termination grace period.

For CLIs, cron jobs and short-lived workers, flush explicitly:

try:
    run_job()
finally:
    tracer_provider.force_flush(timeout_millis=5000)
    tracer_provider.shutdown()
    meter_provider.shutdown()     # also exports the final metric interval
    logger_provider.shutdown()

Sampling

At scale you cannot keep every trace. Head sampling decides when the root span starts, before anything has happened, and the decision rides downstream in the traceparent sampled flag so a trace is kept or dropped whole.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

TracerProvider(resource=resource, sampler=ParentBased(root=TraceIdRatioBased(0.1)))  # keep 10%

Or with environment variables, only if your code passes no sampler:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Things worth knowing:

• Ratio decisions are consistent across services. TraceIdRatioBased derives the decision from the trace ID, so every service using the same ratio makes the same call for the same trace.
• ParentBased is what keeps traces whole. Without it, a downstream service on a lower ratio would drop parts of traces its caller kept.
• An explicit sampler= in code beats the env vars. Code that builds its own sampler and reads only OTEL_TRACES_SAMPLER_ARG silently ignores the sampler name.
• One sampler feeds every exporter. The decision happens before the export fan-out, so you cannot sample 10% to one backend and 100% to another from the same provider.
• Metrics are never sampled. This is a strong argument for putting alerts on metrics rather than on span counts.

The head-sampling trap

Ratio sampling is right for request traffic and wrong for low-volume operations. A webhook that fires thirty times a day, each delivery a fresh root taking its own independent roll at 10%, can produce a whole day with zero traces. The failure path is still visible through error logs and retries, so sampling turns the success path into the only unobservable half of the surface — and “healthy” becomes indistinguishable from “dead endpoint”.

The fix is a custom sampler that exempts named operations:

class KeepCritical(Sampler):
    def __init__(self, ratio, critical):
        self._fallback = ParentBased(TraceIdRatioBased(ratio))
        self._critical = critical

    def should_sample(self, parent_context, trace_id, name, kind=None,
                      attributes=None, links=None, trace_state=None):
        if name in self._critical:
            return SamplingResult(Decision.RECORD_AND_SAMPLE, attributes)
        return self._fallback.should_sample(parent_context, trace_id, name, kind,
                                            attributes, links, trace_state)

Samplers only see what exists at span start: name, kind, links, and attributes passed to start_span. Attributes set later cannot influence the decision.

Tail sampling — “keep every trace with an error or over 2 s, plus 5% of the rest” — needs whole traces in one place, so it runs in a Collector, not in the SDK.

The OpenTelemetry Collector

The Collector (otelcol) is a standalone service that receives telemetry, processes it, and exports it onward. It is optional — everything above exported straight to a backend. It earns its place once you need central control.

A Collector is components wired into pipelines, one per signal:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:            # FIRST: refuse data before the process runs out of memory
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes: {}          # add k8s.pod.name, k8s.namespace.name from the sender's IP
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete       # never let credentials reach storage
      - key: url.query
        action: delete       # pre-signed URLs carry signatures here
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/health"'
  batch:                     # LAST before export: fewer, larger requests
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlp/backend:
    endpoint: ingest.example.com:443
    headers:
      api-key: ${env:BACKEND_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, attributes/scrub, filter/health, batch]
      exporters: [otlp/backend]

Three rules follow from that file:

• A component defined but not listed in a pipeline does nothing. This is the number one Collector debugging mistake.
• Processor order is list order. memory_limiter first, batch last, tail_sampling before batch.
• attributes/scrub is type/instance naming. It lets you use the same component type twice with different settings.

Two distributions exist: core (otelcol, minimal) and contrib (otelcol-contrib, which includes tail_sampling, k8sattributes and most receivers). Most real deployments use contrib.

Tail sampling

processors:
  tail_sampling:
    decision_wait: 10s        # how long to wait for a trace's spans to arrive
    num_traces: 50000         # traces held in memory while waiting
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

A trace is kept if any policy says keep — so you get every error and every slow trace, plus 5% of normal traffic. The catch: all spans of a trace must reach the same collector instance, or each one sees a partial trace and decides wrongly. With multiple replicas you need a first tier using the loadbalancing exporter with routing_key: traceID.

Choosing a backend

OTel gives you a firehose of spans and no way to look at them. The backend is what you actually use day to day: storage, an indexed query engine, waterfall UI, dashboards and alerting.

Free tools worth knowing

Free as in open source, self-hostable:

• Jaeger — traces only, dead simple, one container. The classic first stop.
• SigNoz — traces, metrics and logs plus dashboards and alerting, on ClickHouse.
• Grafana Tempo (traces), Loki (logs), Prometheus/Mimir (metrics) — the most common stack, four moving parts.
• OpenTelemetry Collector itself — useful alone. Its debug exporter prints telemetry to stdout.
• Uptrace, OpenObserve, HyperDX — single-binary alternatives worth a look.

Free tiers if you would rather not self-host: Honeycomb (20M events/month, genuinely generous), Grafana Cloud, New Relic (100 GB/month), SigNoz Cloud trial.

What migration actually costs

The important lesson from anyone who has switched vendors: OTel makes your data portable, not your dashboards.

Moving backends is an environment-variable change, or a dual-emit period if you want to be careful — the instrumentation in your code does not move at all. But dashboards, alert rules and saved queries are vendor-specific and have to be rebuilt by hand. Budget for that, not for the code change.

If you already run Prometheus and Grafana, those are not wasted. See How to run Prometheus with docker and docker-compose and How to run Grafana OSS in Docker and Docker Compose — the Collector has a Prometheus receiver, so existing scrape targets can feed the same pipeline.

Designing telemetry that stays useful

Telemetry is an interface. Dashboards, alerts and saved queries are written against span names and attribute keys exactly the way code is written against function names. Rename one and everything downstream silently returns nothing.

Use the semantic conventions

Semantic conventions (“semconv”) are OTel’s shared dictionary of attribute names. Use them where one exists so that backends can recognise an HTTP call or a database query and render it properly.

Be aware that several names changed as semconv stabilised, and libraries of different ages emit different names for the same thing:

A dashboard filtering http.response.status_code = 404 silently misses every span that recorded http.status_code. When a query “returns nothing”, check which spelling the data actually uses before concluding nothing happened.

Name spans for the operation, not the instance

Define span and attribute names as constants in one telemetry module and import them. A typo in a hard-coded string creates a new, empty operation that no dashboard is watching.

Keep secrets and personal data out

Telemetry is copied to vendors, kept for weeks, and readable by many people. Treat everything you record as semi-public.

Redact at the source. A Collector filter is a safety net, not the plan.

Instrumentation checklist

Use this when adding or reviewing telemetry:

• Span name is a constant, low-cardinality, component.operation
• Attribute keys are constants, namespaced, semconv where one exists
• The span is opened outside the try, so the failure path is inside it
• Every exit path sets the same outcome attribute
• The success path records the same measurements as the failure path
• Caught exceptions call record_exception + set_status(ERROR)
• No secrets, signed URLs or personal content anywhere, including baggage
• Metric attributes are low-cardinality — no IDs
• Work moved to threads carries context, or deliberately starts a linked trace
• Expensive attribute computation is guarded by span.is_recording()
• Short-lived processes flush and shut down providers before exit

On that last performance point: a span costs a few microseconds to create plus memory until export. That is fine for requests, I/O and model calls, and not fine for every iteration of a hot loop. Unsampled spans are nearly free, which is a good reason to sample rather than delete instrumentation.

Troubleshooting

Two debugging tools worth keeping in your pocket:

# print every finished span to stdout, temporarily
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

# see the SDK's own warnings: export failures, dropped attributes, provider overrides
import logging; logging.getLogger("opentelemetry").setLevel(logging.DEBUG)

Related Content

• How to Setup and Run Jaeger With Docker and Docker Compose
• How to run Prometheus with docker and docker-compose
• How to run Grafana OSS in Docker and Docker Compose
• How to run Grafana Loki with docker and docker-compose
• Production-Ready Prometheus on Kubernetes: A Complete Guide

Conclusion

OpenTelemetry is a standard and a set of libraries, not a product. You write instrumentation once against the API, configure an SDK at startup, and ship the data over OTLP to whichever backend you choose — swapping that backend later costs you dashboards, not code.

The working model to carry away:

• A span is one timed operation; a trace is the tree of them; context is what links them, and it breaks on raw thread hops.
• Metrics are cheap and unsampled, so put your alerts there — and keep unbounded IDs off them.
• Logs carry the trace_id of the span they were written in, which is what makes the three signals one investigation instead of three.
• The SDK pipeline is sampler → processors → exporter, and its silences are the hard part: no SDK means no error, a full queue drops spans quietly, and atexit does not run on SIGKILL.
• The Collector is optional until you need central credentials, redaction, tail sampling or vendor fan-out — then it is indispensable.

Start small. Add the instrumentation libraries for your web framework and HTTP client, point them at a local Jaeger container, and look at a real trace from your own application. Almost everyone finds something surprising in the first waterfall they open.