How to run Kafdrop the Kafka Web UI in Docker and Docker compose

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.

Apache Kafka is an open-source platform. Kafka was originally developed by Linkedin and was later incubated as the Apache Project. It can process over 1 million messages per second.

Kafka is an amazing platform for processing a huge number of messages very quickly. However, Kafka has one disadvantage that it does not come with an inbuilt User Interface where the users can see the information related to Kafka.

Kafdrop helps us in solving this problem. It gives us a simple, lightweight, and easy-to-use User Interface where one can not only see the required information but can also create and delete Kafka topics.

Features

View Kafka brokers - topic and partition assignments, and controller status
View topics - partition count, replication status, and custom configuration
Browse messages - JSON, plain text, Avro and Protobuf encoding
View consumer groups - per-partition parked offsets, combined and per-partition lag
Create new topics
View ACLs
Support for Azure Event Hubs

Checkout this also:

Ensure docker is installed and up and running

Before proceeding, please ensure that you have docker up and running in your system. Checkout these guides on how to install docker:

Also install the latest copy of docker-compose for your OS from the releases page here .

Running Kafdrop using docker

Launch container in background:

1
2
3
4
5
docker run -d --rm -p 9080:9000 \
    -e KAFKA_BROKERCONNECT=127.0.0.1:9092 \
    -e JVM_OPTS="-Xms32M -Xmx64M" \
    -e SERVER_SERVLET_CONTEXTPATH="/" \
    obsidiandynamics/kafdrop:latest

The above command will run kafdrop as a docker container expecting kafka to listen on localhost port 9092. You can confirm that it is running as expected by checking docker processes:

1
2
3
$ docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED         STATUS         PORTS                                                             NAMES
c95749e59649   obsidiandynamics/kafdrop:latest    "/kafdrop.sh"            3 minutes ago   Up 3 minutes   0.0.0.0:9080->9000/tcp, :::9080->9000/tcp                         exciting_boy

In the above configuration, I am exposing kafdrop port 9000 to local port 9080. That means I can access kafdrop UI with http://server_ip:9080.

If you have protobuff definitions, use the following:

Launch container in background with protobuff definitions:

1
2
3
4
5
6
docker run -d --rm -v <path_to_protobuff_descriptor_files>:/var/protobuf_desc -p 9000:9000 \
    -e KAFKA_BROKERCONNECT=<host:port,host:port> \
    -e JVM_OPTS="-Xms32M -Xmx64M" \
    -e SERVER_SERVLET_CONTEXTPATH="/" \
    -e CMD_ARGS="--message.format=PROTOBUF --protobufdesc.directory=/var/protobuf_desc" \
    obsidiandynamics/kafdrop

Running Kafdrop with docker-compose

Create a docker-compose.yaml file with this content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
services:
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    ports:
      - 9080:9000
    environment:
      KAFKA_BROKERCONNECT: 127.0.0.1:9092
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
      SERVER_SERVLET_CONTEXTPATH: "/"

  kafka:
    image: obsidiandynamics/kafka:latest
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

To start the services

1
docker-compose up -d

Confirm that they are running with this command:

1
docker-compose ps

Navigating the UI

Open a browser and navigate to http://server_ip:9080/.

The Cluster Overview screen is the landing page of the web UI.

You get to see the overall layout of the cluster the individual brokers that make it up, their addresses and some key broker stats whether they are a controller and the number of partitions each broker owns. The latter is quite important as your cluster size and the number of topics (and therefore partitions) grows, you generally want to see an approximately level distribution of partitions across the cluster.

Next is the Topics List, which in most cases is what you’re really here for. Any reasonably-sized microservices-based ecosystem might have hundreds, if not thousands of topics. As you’d expect, the list is searchable. The stats displayed alongside each topic are fairly ho-hum. The one worth noting is the under-replicated column. Essentially, it’s telling us the number of partition replicas that have fallen behind the primary. Zero is a good figure. Anything else is indicative of either a broker or a network issue that requires immediate attention.

Click on a topic in the list to get to the Topic Overview screen.

The screen is subdivided into four sections.

On the top-left, there is a summary of the topic stats - a handy view, not dissimilar to what you would have seen in the cluster overview.

On the top-right, you can view the custom configuration. In the example above, the topic runs a stock-standard config, so there’s nothing to see. Had the configuration been overridden, you’d see a set of custom values like in the example below.

The bottom-left section enumerates over the partitions. The partition indexes are links - clicking through will reveal the first 100 messages in the topic.

The consumers section on the bottom-right lists the consumer group names as well as their aggregate lag (the sum of all individual partition lags).

Clicking on the consumer group on the Topic Overview gets you into the Consumer View. This screen provides a comprehensive breakdown of a single consumer group.

The view is sectioned by topic. For each topic, a separate table lists the underlying partitions. Against each partition, we see the committed offset, which we can compare against the first and last offsets to see how our consumer is tracking. Conveniently, Kafdrop displays the computed lag for each partition, which is aggregated at the footer of each topic table.

The Message View screen is the coveted topic viewer that has in all likelihood brought you here. You can get to the message view in one of two ways:

Click the View Messages button in the Topic Overview screen.
Click the individual partition link in the Topic Overview.

It’s exactly what you’d expect - a chronologically-ordered list of messages (or records, in Kafka parlance) for a chosen partition.

Each entry conveniently displays the offset, the record key (if one is set), the timestamp of publication, and any headers that may have been appended by the producer.

There’s another little trick up Kafdrop’s sleeve. If the message happens to be a valid JSON document, the topic viewer can nicely format it. Click on the green arrow on the left of the message to expand it.

Conclusion

In this guide we learnt how to run Kafdrop, the Kafka UI using docker and docker compose.