How to run Grafana Loki with docker and docker-compose

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

In this guide, we will learn how to install Grafana Loki and Promtail with Docker and Docker Compose. For production systems, please consider installing Grafana Loki with Tanka or Helm.

The configuration acquired with these installation instructions run Loki as a single binary.

Ensure that docker and docker compose is installed

Since we will be using docker to run our set up, it is important that it is installed and running. Please ensure that you are have docker installed. If you are using an Ubuntu system, checkout this guide on How to Install and Use Docker in Ubuntu 22.04.

Confirm that docker is working as expecting by checking the version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ docker version

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:46 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:00:51 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Next, ensure that docker-compose is installed. Docker compose is available as a python pip package. Ensure that python and pip is installed then install docker-compose with this command:

1
sudo pip3 install docker-compose

This is the version of docker-compose installed in my system:

1
2
3
$ docker-compose --version

docker-compose version 1.29.2, build unknown

Running Grafana in docker

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

In our case, we will need Grafana as a UI to the Loki system. We can use it to query the logs or create specific dashboards based on the log patterns.

To run Grafana with docker, use the following command. We are running the latest version of Grafana OSS - version 9.

1
2
3
docker run --name=grafana \
    -p 3000:3000 \
    grafana/grafana-oss:9.0.2

You can use the -d argument to start in detached mode. Once the server is started, access it by visiting http://server_ip:3000 in your browser.

Running Loki with docker

Loki is a logging backend optimized for users running Prometheus and Kubernetes with great logs search and visualization. Loki works with promtail, an agent which ships the contents of local logs to an instance of Grafana Loki.

First, let us create a directory to store content and use as the current directory.

1
mkdir loki

Next, create a configuration file required to start and run loki. Save these to loki-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Now we can run our loki instance with this command:

1
2
3
4
docker run --name loki \
    -v $(pwd):/config \
    -p 3100:3100 \
    grafana/loki:2.5.0 -config.file=/config/loki-config.yaml

Once loki is running, let us also run promtail in a separate tab to export logs from /var/log/*log to our loki instance. Create a file named promtail-config.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

In a new tab, start promtail with this command:

1
2
3
4
5
docker run --name promtail \
    -v $(pwd):/config \
    -v /var/log:/var/log \
    --link loki \
    grafana/promtail:2.5.0 -config.file=/config/promtail-config.yaml

When both loki and promtail are running, navigate to http://server_ip:3100/metrics to view the metrics and http://server_ip:3100/ready for readiness.

The image is configured to run by default as user loki with UID 10001 and GID 10001. You can use a different user, specially if you are using bind mounts, by specifying the UID with a docker run command and using --user=UID with numeric UID suited to your needs. Root user is UID 1000.

Using docker-compose to run Grafana, Loki and promtail

We can add the instructions above to a docker compose file to make it easy for us to create, update and manage the deployments. Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Add the following to a docker-compose.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
version: '3.9'

networks:
  grafana_loki:

services:
  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - 3000:3000
    networks:
      - grafana_loki

  loki:
    image: grafana/loki:2.5.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - grafana_loki

  promtail:
    image: grafana/promtail:2.5.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - grafana_loki

Save the file in the directory with the configurations then start the services with this command:

1
docker-compose up

Or you can add the -d argument for detached mode. Since we are binding the same ports, you can access in the same way as listed above.

Logging in to Grafana

Login with your admin user (default admin/admin). You will be prompted to change your password the first time you login.

Open side menu (click the Grafana icon in top menu) head to Data Sources and add your data source. Search for loki then add source url to http://loki:3100 (To add our loki instance) then save.

Next go to explore and choose Loki as the data source. To view our var log logs, choose labels, job then varlogs then query.

1
{job="varlogs"} | logfmt | __error__=``

You should see the logs streaming in.

Conclusion

In this guide we managed to run Grafana Loki using docker and docker-compose.