In this guilde we will go through the steps of exporting the logs generated by apps running in a kubernetes cluster into grafana loki.
Grafana Loki is a log aggregation tool inspired by prometheus. It provides you a central place where all your logs can be shipped to. Grafana provides a UI that can be accessed over htttp to check and search your logs.
With logs in Loki you can configure alerting based on set conditions for your log data like when an error occurs.
Installing Loki with Helm
Add the Grafana repository to Helm and update it.
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
Install the loki-stack with Helm:
helm install loki-stack grafana/loki-stack \
--create-namespace \
--namespace loki-stack \
--set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi
# to an existing namespace
helm install loki-stack grafana/loki-stack \
--namespace loki-kip0127 \
--set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi
helm install promtail --namespace loki-kip0127 grafana/promtail -f values.yaml
curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 3127 eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ== logs-prod-us-central1.grafana.net default | kubectl apply --namespace=loki-kip0127 -f -
kubectl get pv,pvc -n loki-kip0127
kubectl get pods -n loki-stack
Sending logs from a standalone host
To send data using Promtail, the simplest approach is to run it with docker.
Replace
Save it to promtail/config.yaml.
server:
http_listen_port: 0
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
client:
url: https://3127:eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ==@logs-prod-us-central1.grafana.net/api/prom/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*.log
Run promtail via docker:
docker run --name promtail --volume "$PWD/promtail:/etc/promtail" --volume "/var/log:/var/log" grafana/promtail:master -config.file=/etc/promtail/config.yaml
Simple:
curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 3127 eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ== logs-prod-us-central1.grafana.net default | kubectl apply --namespace=default -f -
Replace
Sending logs using helm
Adding Promtail DaemonSet
To ship all your pods logs, we’re going to set up Promtail as a DaemonSet in our cluster. This means it will run on each node of the cluster. We’ll then configure it to find the logs of your containers on the host.
Promtail uses the same service discovery as Prometheus
Make sure the scrape_configs of Promtail matches the Prometheus one. Not only this is simpler to configure, but this also means Metrics and Logs will have the same metadata (labels) attached by the Prometheus service discovery. Then, when querying Grafana, you’ll be able to correlate metrics and logs very quickly. (Read more about that here.
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo
Create loki values:
cat > values.yaml <<EOF
loki:
serviceName: "logs-prod-us-central1.grafana.net"
servicePort: 443
serviceScheme: https
user: 3127
password: eyJrIjoiMmQwMDUxMGJmZjhlZWIyMzIyYjU3YzgwMzM4ZjIwNDllZmZiYmMxMiIsIm4iOiJva3RldG8iLCJpZCI6MjY1ODI4fQ==
EOF
Once you’re ready let’s create a new namespace monitoring and add Promtail to it:
kubectl create namespace monitoring
helm install promtail --namespace monitoring grafana/promtail -f values.yaml
kubectl get -n monitoring pods
You can reach your Grafana instance and start exploring your logs. For example, if you want to see all logs in the monitoring namespace use {namespace="monitoring"}. You can also expand a single log line to discover all labels available from the Kubernetes service discovery.
1 Comment
Pingback: How to Setup Promtail, Grafana and Loki for free Log Management in Debian 11