Grafana Loki is an open source log aggregation tool provided by the Grafana Labs. It is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. This reduces the workload of processing and storing logs.
Promtail is Loki’s log collector, which sends log tags to Grafana Loki for indexing.
Related posts:
# Installing Grafana
Ensure the system is up to date before proceeding:
| |
Add the Grafana GPG key:
| |
Add Grafana APT repository:
| |
Next update apt repo then install grafana:
| |
Confirm installed grafana version
| |
Finally start and enable Grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-serverGrafana is now installed and can be accessed through the server’s IP and port 3000. (https://server_IP:3000)
You need to allow port 3000 to pass through the firewall if you have firewall enabled. Use this command:
sudo ufw allow proto tcp from any to any port 3000# Install Grafana Loki
Loki is available as a Linux binary on github release page for the app here. Scroll down to the Assets section under the version that you want to install. Download using this command:
curl -LO https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zipExtract the downloaded zip file. You need the unzip program installed. Ensure that it is present using this command:
sudo apt install -y unzipYou can then extract the file and move it to the binary directory:
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/lokiCreate config directory and add the configuration files provided by Grafana Labs:
sudo mkdir /etc/loki
sudo mkdir -p /data/lokiThen edit the config file
sudo vim /etc/loki/config.yamlAdd this content:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24hNext create a loki systemd service. Use this command to create file
/etc/systemd/system/loki.service.
sudo tee /etc/systemd/system/loki.service<Reload the system daemon, and then start the Loki service:
sudo systemctl daemon-reload
sudo systemctl start lokiYou can check whether the service has started successfully:
$ sudo systemctl daemon-reload
sudo systemctl start loki
admin@dev-debiansrv:~/tmp$ sudo systemctl status loki
● loki.service - Loki service
Loaded: loaded (/etc/systemd/system/loki.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2022-01-20 18:39:08 UTC; 19s ago
Main PID: 34374 (loki)
Tasks: 8 (limit: 4626)
Memory: 17.8M
CPU: 111ms
CGroup: /system.slice/loki.service
└─34374 /usr/local/bin/loki -config.file /etc/loki/config.yaml
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.487479168Z caller=compactor.go:263 msg="compactor is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619449446Z caller=scheduler.go:629 msg="scheduler is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619551917Z caller=module_service.go:64 msg=initialising module=query-frontend
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619815925Z caller=module_service.go:64 msg=initialising module=querier
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619923866Z caller=loki.go:355 msg="Loki started"
Jan 20 18:39:11 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:11.620368148Z caller=worker.go:205 msg="adding connection" addr=127.0.0.1:9096You can now access Loki indicators by https://server-IP:3100/metrics
# Installing Promtail Agent
Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.
Download the Promtail binary from the Loki github releases page. Scroll down to the Assets section under the version that you want to install. Download using this command:
curl -LO https://github.com/grafana/loki/releases/download/v2.4.2/promtail-linux-amd64.zipExtract the downloaded zip file and move the binary to executable directory.
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtailConfirm the installed version
$ promtail --version
promtail, version 2.4.2 (branch: HEAD, revision: 525040a32)
build user: root@5d9e7a4c92e6
build date: 2022-01-12T16:48:53Z
go version: go1.16.2
platform: linux/amd64Next create config directory and a data directory:
sudo mkdir /etc/promtail
sudo mkdir -p /data/promtailCreate a YAML configuration file for Promtail:
| |
Add this content to the file:
| |
Next, create a systemd service to manage the promtail service:
| |
Reload and start the Promtail service
| |
Confirm whether the service is running:
| |
We have managed to install Grafana, Loki and promtail.
# Configuring Loki Data Source
Now that everything is set up, we need to add loki in the Grafana UI. Log in to the Grafana web interface and click on Configuration -> Data Sources. Then click Add data source. Search for Loki and add these values:
Enter the following values for Loki:
| |
Scroll to the bottom then Click Save and test. You should see a notification that the data source has been successfully added.
# Visualize Loki logs on Grafana
With everything set up, we can use Grafana to visualize the logs. Click on explore Then choose Loki In the data source. In the log browser, search for {job="varlogs"} to search var logs messages:
That is it! We have successfully installed Grafana Loki with Promtail and can visualize logs on the Grafana dashboard.