How to Setup Promtail, Grafana and Loki for free Log Management in Debian 11

Grafana Loki is an open source log aggregation tool provided by the Grafana Labs. It is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. This reduces the workload of processing and storing logs.

Promtail is Loki’s log collector, which sends log tags to Grafana Loki for indexing.

Installing Grafana

Ensure the system is up to date before proceeding:

1
2
sudo apt update
sudo apt upgrade -y

Add the Grafana GPG key:

1
2
sudo apt-get install -y gnupg2 curl
curl https://packages.grafana.com/gpg.key | sudo apt-key add -

Add Grafana APT repository:

1
2
sudo apt install software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Next update apt repo then install grafana:

1
sudo apt update && sudo apt install -y grafana

Confirm installed grafana version

1
2
3
4
5
6
7
grafana:
  Installed: 8.3.4
  Candidate: 8.3.4
  Version table:
 *** 8.3.4 500
        500 https://packages.grafana.com/oss/deb stable/main amd64 Packages
        100 /var/lib/dpkg/status

Finally start and enable Grafana

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Grafana is now installed and can be accessed through the server’s IP and port 3000. (https://server_IP:3000)

You need to allow port 3000 to pass through the firewall if you have firewall enabled. Use this command:

sudo ufw allow proto tcp from any to any port 3000

Install Grafana Loki

Loki is available as a Linux binary on github release page for the app here. Scroll down to the Assets section under the version that you want to install. Download using this command:

curl -LO https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip

Extract the downloaded zip file. You need the unzip program installed. Ensure that it is present using this command:

sudo apt install -y unzip

You can then extract the file and move it to the binary directory:

unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki

Create config directory and add the configuration files provided by Grafana Labs:

sudo mkdir /etc/loki
sudo mkdir -p /data/loki

Then edit the config file

sudo vim /etc/loki/config.yaml

Add this content:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /data/loki
  storage:
    filesystem:
      chunks_directory: /data/loki/chunks
      rules_directory: /data/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Next create a loki systemd service. Use this command to create file

/etc/systemd/system/loki.service.

sudo tee /etc/systemd/system/loki.service<

Reload the system daemon, and then start the Loki service:

sudo systemctl daemon-reload
sudo systemctl start loki

You can check whether the service has started successfully:

$ sudo systemctl daemon-reload
sudo systemctl start loki
admin@dev-debiansrv:~/tmp$ sudo systemctl status loki
● loki.service - Loki service
     Loaded: loaded (/etc/systemd/system/loki.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-01-20 18:39:08 UTC; 19s ago
   Main PID: 34374 (loki)
      Tasks: 8 (limit: 4626)
     Memory: 17.8M
        CPU: 111ms
     CGroup: /system.slice/loki.service
             └─34374 /usr/local/bin/loki -config.file /etc/loki/config.yaml

Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.487479168Z caller=compactor.go:263 msg="compactor is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619449446Z caller=scheduler.go:629 msg="scheduler is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619551917Z caller=module_service.go:64 msg=initialising module=query-frontend
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619815925Z caller=module_service.go:64 msg=initialising module=querier
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619923866Z caller=loki.go:355 msg="Loki started"
Jan 20 18:39:11 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:11.620368148Z caller=worker.go:205 msg="adding connection" addr=127.0.0.1:9096

You can now access Loki indicators by https://server-IP:3100/metrics

`Installing Promtail Agent`

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.

Download the Promtail binary from the Loki github releases page. Scroll down to the Assets section under the version that you want to install. Download using this command:

curl -LO https://github.com/grafana/loki/releases/download/v2.4.2/promtail-linux-amd64.zip

Extract the downloaded zip file and move the binary to executable directory.

unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

Confirm the installed version

$ promtail --version
promtail, version 2.4.2 (branch: HEAD, revision: 525040a32)
  build user:       root@5d9e7a4c92e6
  build date:       2022-01-12T16:48:53Z
  go version:       go1.16.2
  platform:         linux/amd64

Next create config directory and a data directory:

sudo mkdir /etc/promtail
sudo mkdir -p /data/promtail

Create a YAML configuration file for Promtail:

1
sudo vim /etc/promtail/config.yaml

Add this content to the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /data/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: <meta charset="utf-8">grafanalogs
  static_configs:
  - targets:
      - localhost
    labels:
      job: <meta charset="utf-8">grafana
      __path__: /var/log/grafana/grafana.log

Next, create a systemd service to manage the promtail service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
sudo tee /etc/systemd/system/promtail.service<<EOF
&#91;Unit]
Description=Promtail service
After=network.target

&#91;Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/config.yaml

&#91;Install]
WantedBy=multi-user.target
EOF

Reload and start the Promtail service

1
2
sudo systemctl daemon-reload
sudo systemctl start promtail.service

Confirm whether the service is running:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
$ systemctl status promtail
&#x25CF; promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-01-20 18:51:10 UTC; 12s ago
   Main PID: 34488 (promtail)
      Tasks: 9 (limit: 4626)
     Memory: 29.9M
        CPU: 532ms
     CGroup: /system.slice/promtail.service
             &#x2514;&#x2500;34488 /usr/local/bin/promtail -config.file /etc/promtail/config.yaml

Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: ts=2022-01-20T18:51:15.303156121Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: level=info ts=2022-01-20T18:51:15.303172481Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: ts=2022-01-20T18:51:15.303201692Z caller=log.go:168 level=info msg="Seeked /var/log/kern.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: level=info ts=2022-01-20T18:51:15.303214345Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/kern.log
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: ts=2022-01-20T18:51:15.303238377Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: level=info ts=2022-01-20T18:51:15.305348131Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/lastlog
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: ts=2022-01-20T18:51:15.30581234Z caller=log.go:168 level=info msg="Seeked /var/log/syslog - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: level=info ts=2022-01-20T18:51:15.305873583Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/syslog
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: ts=2022-01-20T18:51:15.306786457Z caller=log.go:168 level=info msg="Seeked /var/log/user.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail&#91;34488]: level=info ts=2022-01-20T18:51:15.307060365Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/user.log

We have managed to install Grafana, Loki and promtail.

`Configuring Loki Data Source`

Now that everything is set up, we need to add loki in the Grafana UI. Log in to the Grafana web interface and click on Configuration -> Data Sources. Then click Add data source. Search for Loki and add these values:

Enter the following values for Loki:

1
2
Name: Loki
URL: https://127.0.0.1:3100

Scroll to the bottom then Click Save and test. You should see a notification that the data source has been successfully added.

`Visualize Loki logs on Grafana`

With everything set up, we can use Grafana to visualize the logs. Click on explore Then choose Loki In the data source. In the log browser, search for {job="varlogs"} to search var logs messages:

That is it! We have successfully installed Grafana Loki with Promtail and can visualize logs on the Grafana dashboard.