How to Export Kubernetes Logs to Grafana Loki with Helm

Export Kubernetes pod logs to Grafana Loki with Helm: ship them with a Grafana Alloy DaemonSet or the legacy Promtail chart, then query them in Grafana.

In this guide we will go through the steps of exporting the logs generated by apps running in a kubernetes cluster into grafana loki.

Grafana Loki is a log aggregation tool inspired by prometheus. It provides you a central place where all your logs can be shipped to. Grafana provides a UI that can be accessed over https to check and search your logs.

With logs in Loki you can configure alerting based on set conditions for your log data like when an error occurs.

Promtail is end of life. Grafana declared Promtail EOL on March 2, 2026; it gets no more updates, and all new development happens in Grafana Alloy. The grafana/promtail and grafana/loki-stack Helm charts used below are both marked deprecated. For a new cluster, jump to Shipping logs with Grafana Alloy. The Promtail sections are kept for clusters that still run it. To move an existing Promtail config over, see the official Promtail-to-Alloy migration guide.

Installing Loki with Helm

Add the Grafana repository to Helm and update it.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install the loki-stack with Helm:

helm install loki-stack grafana/loki-stack \
    --create-namespace \
    --namespace loki-stack \
    --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi

To install in an existing namespace

helm install loki-stack grafana/loki-stack \
    --namespace <namespace> \
    --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi

With promtail.enabled=true, loki-stack already deploys Promtail as a DaemonSet next to Loki, so you do not need to install it separately.

The loki-stack chart is deprecated and pinned to Loki 2.9. For a current Loki install, use the grafana/loki chart as shown in How to Run Grafana Loki with Helm and Kustomize in Kubernetes.

Ensure that the application is running:

kubectl get pv,pvc -n loki-stack

kubectl get pods -n loki-stack

Sending logs from a standalone host

To send data using Promtail, the simplest approach is to run it with docker.

The examples below push to Grafana Cloud Logs. Replace <INSTANCE_ID> with your Loki instance ID (the user), <API_KEY> with a Grafana Cloud access policy token that has the logs:write scope, and <LOKI_HOST> with your Loki host (for example logs-prod-us-central1.grafana.net). All three are on your stack’s Loki details page. Save it to promtail/config.yaml.

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://<INSTANCE_ID>:<API_KEY>@<LOKI_HOST>/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

Run promtail via docker:

docker run --name promtail --volume "$PWD/promtail:/etc/promtail" --volume "/var/log:/var/log" grafana/promtail:3.6.11 -config.file=/etc/promtail/config.yaml

To run Promtail in a Kubernetes cluster without Helm, Loki ships a script that renders a Promtail DaemonSet manifest which you can pipe into kubectl:

curl -fsS https://raw.githubusercontent.com/grafana/loki/main/tools/promtail.sh | sh -s <INSTANCE_ID> <API_KEY> <LOKI_HOST> default | kubectl apply --namespace=default -f  -

Sending logs using helm

Adding Promtail DaemonSet

To ship all your pods logs, we’re going to set up Promtail as a DaemonSet in our cluster. This means it will run on each node of the cluster. We’ll then configure it to find the logs of your containers on the host.

Promtail uses the same service discovery as Prometheus

Make sure the scrape_configs of Promtail matches the Prometheus one. Not only this is simpler to configure, but this also means Metrics and Logs will have the same metadata (labels) attached by the Prometheus service discovery. Then, when querying Grafana, you’ll be able to correlate metrics and logs very quickly.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo grafana/promtail

Create promtail values. Current versions of the chart take a list of Loki push endpoints under config.clients:

cat > values.yaml <<EOF
config:
  clients:
    - url: https://<LOKI_HOST>/loki/api/v1/push
      basic_auth:
        username: "<INSTANCE_ID>"
        password: "<API_KEY>"
EOF

For a Loki running inside the cluster, point url at its service instead (for example http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push) and drop basic_auth.

Once you’re ready let’s create a new namespace monitoring and add Promtail to it:

kubectl create namespace monitoring
helm install promtail --namespace monitoring grafana/promtail -f values.yaml

kubectl get -n monitoring pods

Shipping logs with Grafana Alloy

Grafana Alloy replaces Promtail. Its Helm chart deploys Alloy as a DaemonSet by default, and you give it the pipeline in alloy.configMap.content. The config below follows the official Kubernetes logs example: it discovers pods on each node, adds namespace, pod, container and app labels, tails the logs through the Kubernetes API, and writes them to Loki.

cat > alloy-values.yaml <<'EOF'
alloy:
  configMap:
    content: |
      discovery.kubernetes "pod" {
        role = "pod"
        selectors {
          role  = "pod"
          field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
        }
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "https://<LOKI_HOST>/loki/api/v1/push"
          basic_auth {
            username = "<INSTANCE_ID>"
            password = "<API_KEY>"
          }
        }
      }
EOF

helm install alloy grafana/alloy --namespace monitoring --create-namespace -f alloy-values.yaml

kubectl get -n monitoring pods

As with Promtail, use the in-cluster Loki service URL and drop the basic_auth block when Loki runs in the same cluster. In production, keep the token in a Kubernetes Secret rather than in the values file.

Querying the logs in Grafana

You can reach your Grafana instance and start exploring your logs. For example, if you want to see all logs in the monitoring namespace use {namespace="monitoring"}. You can also expand a single log line to discover all labels available from the Kubernetes service discovery. See the LogQL documentation for filtering and aggregating queries.

comments powered by Disqus
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy