How to export Kubernetes logs to Grafana Loki

In this guilde we will go through the steps of exporting the logs generated by apps running in a kubernetes cluster into grafana loki.

Grafana Loki is a log aggregation tool inspired by prometheus. It provides you a central place where all your logs can be shipped to. Grafana provides a UI that can be accessed over htttp to check and search your logs.

With logs in Loki you can configure alerting based on set conditions for your log data like when an error occurs.

# Installing Loki with Helm

Add the Grafana repository to Helm and update it.

1
2
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install the loki-stack with Helm:

1
2
3
4
helm install loki-stack grafana/loki-stack \
    --create-namespace \
    --namespace loki-stack \
    --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi

To install in an existing namespace

1
2
3
helm install loki-stack grafana/loki-stack \
    --namespace loki-kip0127 \
    --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi

Install promtail

1
2
3
helm install promtail --namespace loki-kip0127 grafana/promtail -f values.yaml

curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 3127 eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ== logs-prod-us-central1.grafana.net default | kubectl apply --namespace=loki-kip0127 -f  -

Ensure that the application is running:

1
2
3
kubectl get pv,pvc -n loki-kip0127

kubectl get pods -n loki-stack

# Sending logs from a standalone host

To send data using Promtail, the simplest approach is to run it with docker.

Replace in the config below with a Grafana.com API key with the MetricsPublisher role. Generate now Save it to promtail/config.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://3127:eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ==@logs-prod-us-central1.grafana.net/api/prom/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

Run promtail via docker:

1
docker run --name promtail --volume "$PWD/promtail:/etc/promtail" --volume "/var/log:/var/log" grafana/promtail:master -config.file=/etc/promtail/config.yaml

A simpler way to do it will be using curl:

1
curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 3127 eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ== logs-prod-us-central1.grafana.net default | kubectl apply --namespace=default -f  -

Replace with a Grafana.com API key with the MetricsPublisher role.

# Sending logs using helm

Adding Promtail DaemonSet

To ship all your pods logs, we’re going to set up Promtail as a DaemonSet in our cluster. This means it will run on each node of the cluster. We’ll then configure it to find the logs of your containers on the host.

Promtail uses the same service discovery as Prometheus

Make sure the scrape_configs of Promtail matches the Prometheus one. Not only this is simpler to configure, but this also means Metrics and Logs will have the same metadata (labels) attached by the Prometheus service discovery. Then, when querying Grafana, you’ll be able to correlate metrics and logs very quickly. (Read more about that here.

1
2
3
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo

Create loki values:

1
2
3
4
5
6
7
8
cat > values.yaml <<EOF
loki:
  serviceName: "logs-prod-us-central1.grafana.net"
  servicePort: 443
  serviceScheme: https
  user: 3127
  password: eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ==
EOF

Once you’re ready let’s create a new namespace monitoring and add Promtail to it:

1
2
3
4
kubectl create namespace monitoring
helm install promtail --namespace monitoring grafana/promtail -f values.yaml

kubectl get -n monitoring pods

You can reach your Grafana instance and start exploring your logs. For example, if you want to see all logs in the monitoring namespace use {namespace=“monitoring”}. You can also expand a single log line to discover all labels available from the Kubernetes service discovery.