How to Setup and Run Jaeger With Docker and Docker Compose

Jaeger is a distributed tracing system. Distributed tracing has become essential for developers working with microservices architectures. Jaeger, an open-source distributed tracing system, offers a powerful solution for monitoring and troubleshooting complex distributed systems.

Jaeger is an open-source distributed tracing system that helps you monitor and troubleshoot transactions in complex distributed systems. It allows you to track requests as they flow through your microservices architecture, providing visibility into the performance and behavior of your applications.

In simple termas, Jaeger is a tracing tool that sheds light on how requests flow across your services.

Distributed tracing is crucial in microservices environments because it allows you to:

Identify performance bottlenecks
Debug and troubleshoot issues across services
Understand the flow of requests through your system
Optimize your application’s overall performance

Jaeger’s compatibility with OpenTelemetry — a collection of tools, APIs, and SDKs for instrumenting, generating, collecting, and exporting telemetry data — makes it an even more powerful choice for developers. This compatibility ensures that you can easily integrate Jaeger with a wide range of applications and services.

While Jaeger does add some overhead to your application, it’s designed to be lightweight. The impact can be minimized through proper sampling strategies and configuration. In most cases, the performance impact is negligible compared to the benefits of distributed tracing.

With Jaeger you can:

Monitor and troubleshoot distributed workflows
Identify performance bottlenecks
Track down root causes
Analyze service dependencies

How is Jaeger different from other tracing systems like Zipkin?

Jaeger and Zipkin are both open-source distributed tracing systems, but Jaeger offers more advanced features like adaptive sampling and a more scalable architecture. Jaeger also has better support for OpenTelemetry, making it more future-proof.

This guide will walk you through the process of implementing Jaeger, from setup to advanced usage.

Setting Up Jaeger

There is an all in one container image provided by the team jaegertracing/jaeger includes all the necessary components to get started.

You need to have docker installed in your system. Confirm that is it running using this command:

1
2
3
$ docker --version

Docker version 27.4.1, build b9d17ea

Pull the Jaeger All-in-One image:

1
docker pull jaegertracing/all-in-one:1.57

Run the Jaeger All-in-One container:

1
2
3
4
5
6
7
docker run --rm --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 5778:5778 \
  -p 9411:9411 \
  jaegertracing/jaeger:2.2.0

You can use docker compose to achieve the same

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
services:
  jaeger:
    image: jaegertracing/jaeger:2.2.0
    container_name: jaeger
    ports:
      - "16686:16686" # Jaeger UI
      - "4317:4317"   # OpenTelemetry gRPC
      - "4318:4318"   # OpenTelemetry HTTP
      - "5778:5778"   # Configuration service
      - "9411:9411"   # Zipkin compatibility
    restart: unless-stopped

This runs the all-in-one configuration of Jaeger that combines collector and query components in a single process and uses a transient in-memory storage for trace data with all necessary ports exposed.

Access the Jaeger UI by opening a web browser and navigating to http://localhost:16686.
Verify the installation by sending test spans. You can use the Jaeger client libraries or OpenTelemetry SDK to instrument a simple application and generate traces.

Jaeger User Interface

To access the Jaeger UI, navigate to http://localhost:16686 in your web browser. The UI provides a comprehensive view of the tracing data, including:

Service Map: A visual representation of the services and their interactions.
Tracing: A detailed view of individual traces, including the request and response data.
Span: A detailed view of individual spans, including the request and response data.
Query: A search interface for querying tracing data.
Ports Exposed by the Jaeger Docker Container

The Jaeger Docker container exposes the following ports:

6831: UDP port for sending tracing data.
6832: UDP port for sending tracing data.
5778: HTTP port for the Jaeger UI.
16686: HTTP port for the Jaeger UI.
4317: HTTP port for the Jaeger UI.
4318: HTTP port for the Jaeger UI.
14250: HTTP port for the Jaeger UI.
14268: HTTP port for the Jaeger UI.
9411: HTTP port for the Jaeger UI.
These ports are used to communicate between the Jaeger components and access the Jaeger UI.

Jaeger Architecture

Jaeger’s architecture consists of several key components:

Client Libraries: These libraries are used to instrument your application code. They create spans and send them to the Jaeger Agent.
Agent: A network daemon that listens for spans sent by the client libraries. It batches and sends them to the Collector.
Collector: Receives traces from the Agent and runs them through a processing pipeline. It then stores them in a storage backend.
Query: A service that retrieves traces from storage and hosts a UI to display them.
UI: A web interface for searching and analyzing traces.

Data flows from your instrumented application through the client libraries to the Agent, then to the Collector, and finally to storage. The Query service retrieves this data from storage to display in the UI.

Jaeger supports multiple storage options, including Cassandra, Elasticsearch, and in-memory storage (for development). The choice of storage depends on your scalability needs and existing infrastructure.

Sampling plays a crucial role in Jaeger’s architecture. It allows you to control the amount of tracing data you collect, which is essential for managing performance and storage costs in high-traffic systems.

Instrumenting application with Jaeger

To instrument your application, you can choose between Jaeger’s native client libraries or the OpenTelemetry SDK. Here’s an example of basic tracing using the Jaeger Python client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
from jaeger_client import Config

def init_tracer(service):
    config = Config(
        config={
            'sampler': {
                'type': 'const',
                'param': 1,
            },
            'logging': True,
        },
        service_name=service,
    )
    return config.initialize_tracer()

tracer = init_tracer('my-service')

with tracer.start_span('TestSpan') as span:
    span.set_tag('hello', 'world')
    # Your code here

This code initializes a tracer and creates a simple span with a tag. To propagate context across service boundaries, you’ll need to pass the SpanContext between services. This is typically done by injecting the context into HTTP headers or message queue headers.

Advanced Instrumentation Techniques

These are some advanced techniques that you can instrument with jaeger:

Custom Samplers: Create samplers that make intelligent decisions about which traces to sample based on your specific needs.
Baggage: Use baggage to pass data along the entire trace, which can be useful for correlating information across services.
Multiple Spans: Create and manage multiple spans within a single trace to represent different operations or sub-operations.
Logging Integration: Integrate Jaeger with your logging system to enhance debugging capabilities.

Jaeger in a Production Environment

When deploying Jaeger in production, consider the following:

Scalability: Each component (Agent, Collector, Query) can be scaled independently. Use load balancers to distribute traffic.
Storage: Implement a production-ready storage backend like Elasticsearch or Cassandra. Ensure proper sizing and configuration for your expected data volume.
Security: Set up secure communication between Jaeger components using TLS. Implement authentication and authorization for the Jaeger UI.
Sampling: Implement appropriate sampling strategies based on your traffic patterns and tracing needs. Dynamic sampling can help balance data collection and system performance.

Analyzing Traces with Jaeger UI

The Jaeger UI provides powerful tools for analyzing traces:

Use the search functionality to find relevant traces based on service, operation, tags, or duration.
Examine the trace timeline to understand the relationships between spans and identify long-running operations.
Inspect span details, including tags and logs, to gather context about each operation.
Use the comparison view to analyze multiple traces side by side and identify patterns or anomalies.

Best Practices for Effective Jaeger Implementation

To get the most out of Jaeger:

Follow consistent naming conventions for services and operations to make searching and filtering easier.
Implement appropriate sampling strategies to balance data collection and system performance.
Ensure proper error handling and logging within instrumented code to provide context for issues.
Regularly review and optimize your tracing implementation to ensure it continues to meet your needs as your system evolves.

Service Performance Monitoring (SPM) with Jaeger

Service Performance Monitoring (SPM) is a feature in Jaeger that provides service-level and operation-level aggregation of key metrics like request rates, error rates, and durations (latencies). It helps identify interesting traces (e.g. high QPS, slow or erroneous requests) without needing to know the service or operation names up-front.

How does SPM work?

SPM works by aggregating span data from Jaeger to produce RED (Request, Error, Duration) metrics. The OpenTelemetry Collector’s Span Metrics Processor is used to generate these metrics from the incoming traces.

Key features of SPM

Service-level and operation-level aggregation of request rates, error rates, and latencies (P95, P75, P50)
“Impact” metric computed as the product of latency and request rate to highlight high-impact operations
Pre-populated Trace search with relevant service, operation and lookback period for interesting traces

Accessing SPM in Jaeger

The SPM feature can be accessed from the “Monitor” tab in the Jaeger UI. It requires Jaeger 1.57 or later.

Limitations

SPM is still an experimental feature and may have further changes in the future Requires OpenTelemetry Collector 0.63.1 or later with the Span Metrics Processor enabled By providing aggregated RED metrics and highlighting high-impact traces, SPM makes it easier to monitor service performance and identify issues in complex distributed systems using Jaeger.

Conclusion

Jaeger is a powerful open-source tool for distributed tracing in microservices architectures. Proper setup and instrumentation are crucial for effective use of Jaeger. Understanding Jaeger’s architecture helps in optimizing its deployment. Analyzing traces through the Jaeger UI can significantly improve system performance and debugging capabilities.