How to Run Grafana Loki with Docker and Docker Compose

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate: instead of indexing full log contents, Loki indexes a set of labels for each log stream.

In this guide, we will learn how to install Grafana Loki and Promtail with Docker and Docker Compose. For production systems, please consider installing Grafana Loki with Tanka or Helm.

The configuration in this guide runs Loki as a single binary (great for local/dev). For production, consider Loki distributed mode and deploy with Helm or Tanka.

Ensure Docker and Docker Compose are installed

Since we will be using Docker to run our setup, it is important that it is installed and working. If you are using Ubuntu, check out How to Install and Use Docker in Ubuntu 22.04.

Confirm that Docker is working as expected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ docker version

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:46 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:00:51 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Compose is now included with Docker Desktop and is available on Linux as a Docker plugin. Confirm it’s available with:

1
docker compose version

Running Grafana in Docker

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

In our case, we will need Grafana as a UI to the Loki system. We can use it to query the logs or create specific dashboards based on the log patterns.

To run Grafana with docker, use the following command. We are running the latest version of Grafana OSS - version 9.

1
2
3
docker run -d --name grafana \
  -p 3000:3000 \
  grafana/grafana-oss:9.0.2

You can use the -d argument to start in detached mode. Once the server is started, access it by visiting http://server_ip:3000 in your browser.

Running Loki with Docker

Loki is a logging backend optimized for users running Prometheus and Kubernetes with great logs search and visualization. Loki works with promtail, an agent which ships the contents of local logs to an instance of Grafana Loki.

First, create a working directory:

1
mkdir -p loki && cd loki

Next, create a configuration file required to start and run Loki. Save this as loki-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Now we can run our loki instance with this command:

1
2
3
4
5
6
7
8
docker network create grafana_loki || true

docker run -d --name loki \
  --network grafana_loki \
  -v "$(pwd)/loki-config.yaml":/etc/loki/config.yaml \
  -v loki_data:/loki \
  -p 3100:3100 \
  grafana/loki:2.5.0 -config.file=/etc/loki/config.yaml

Once Loki is running, run Promtail to ship logs from /var/log/*log to Loki. Create a file named promtail-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

Start promtail with this command:

1
2
3
4
5
6
docker run -d --name promtail \
  --network grafana_loki \
  -v "$(pwd)/promtail-config.yaml":/etc/promtail/config.yml \
  -v /var/log:/var/log:ro \
  -v promtail_positions:/positions \
  grafana/promtail:2.5.0 -config.file=/etc/promtail/config.yml

When both loki and promtail are running, navigate to http://server_ip:3100/metrics to view the metrics and http://server_ip:3100/ready for readiness.

Note: some Loki/Promtail images run as a non-root user by default. If you use bind mounts and hit permissions issues, either adjust directory permissions or run the container with --user (root is UID 0).

Using Docker Compose to run Grafana, Loki and Promtail

We can add the instructions above to a Docker Compose file to make it easy to create, update and manage the deployments. Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Add the following to a docker-compose.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
version: "3.9"
networks:
  grafana_loki:

volumes:
  loki_data:
  promtail_positions:
  grafana_data:

services:
  grafana:
    image: grafana/grafana-oss:9.0.2
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - grafana_loki

  loki:
    image: grafana/loki:2.5.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/config.yaml
    networks:
      - grafana_loki

  promtail:
    image: grafana/promtail:2.5.0
    volumes:
      - /var/log:/var/log:ro
      - ./promtail-config.yaml:/etc/promtail/config.yml
      - promtail_positions:/positions
    command: -config.file=/etc/promtail/config.yml
    networks:
      - grafana_loki

Save the file in the directory with the configurations then start the services with this command:

1
docker compose up -d

Or you can add the -d argument for detached mode. Since we are binding the same ports, you can access in the same way as listed above.

Logging in to Grafana

Login with your admin user (default admin/admin). You will be prompted to change your password the first time you login.

Open the side menu (Grafana icon) → Connections → Data sources → Add data source → choose Loki. Set the URL to http://loki:3100 and click Save & test.

Next go to Explore and choose Loki as the data source. To view /var/log/*log logs, choose labels → job → varlogs then run a query.

1
{job="varlogs"} | logfmt | __error__=``

You should see the logs streaming in.

Conclusion

In this guide we managed to run Grafana, Loki, and Promtail using Docker and Docker Compose, then query logs from Grafana Explore.