Containers Kubernetes Monitoring

How to Setup Kube State Metrics on Kubernetes

Step-by-step guide to installing kube-state-metrics on Kubernetes. Deploy with manifests or Helm, configure Prometheus scraping, and expose metrics for deployments, pods, nodes, and more.

kube-state-metrics is a service that watches the Kubernetes API and exposes metrics about the state of objects in your cluster (e.g. pod status, node capacity, deployment replicas, resource requests and limits). It does not report resource usage (CPU/memory usage); that is provided by the Kubernetes metrics server and is used for scaling (e.g. HPA). kube-state-metrics focuses on object state (e.g. desired vs. available replicas, pod phase, node conditions), which is useful for alerting and dashboards in Prometheus and Grafana.

In this guide you will:

  • Understand what kube-state-metrics provides and how it fits with Prometheus
  • Install kube-state-metrics using manifests (ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, Service) or Helm
  • Optionally tune resources and limit exposed metrics
  • Add a Prometheus scrape job so Prometheus can collect the metrics

Prerequisites: A Kubernetes cluster (1.20+) with kubectl configured. For Prometheus integration, Prometheus must be installed (e.g. How to Setup Prometheus Monitoring on Kubernetes).

What kube-state-metrics Provides

kube-state-metrics exposes metrics on an HTTP endpoint in Prometheus exposition format. Prometheus scrapes this endpoint and stores the metrics; you can then query them with PromQL and visualize them in Grafana.

Examples of metrics (see official docs for the full list):

  • Nodes: status, capacity (CPU, memory), allocatable
  • Pods: status (waiting, running, ready), phase, restarts
  • Deployments / ReplicaSets: desired, available, unavailable, updated replicas
  • StatefulSets / DaemonSets: similar replica and status metrics
  • Jobs / CronJobs: active, failed, succeeded, completion time
  • Ingress: info, backend metrics
  • PersistentVolumes / PersistentVolumeClaims: status, capacity, phase
  • Resource requests and limits (from pod specs)
  • HorizontalPodAutoscalers, PodDisruptionBudgets, and other API objects

Benefits:

  • Metrics at an HTTP endpoint (e.g. http://kube-state-metrics:8080/metrics)
  • Prometheus exposition format, so Prometheus can scrape without extra conversion
  • Raw data derived from the Kubernetes API (list/watch)

You can open the /metrics URL in a browser, but the output is plain text; Prometheus and Grafana make it queryable and visual.

Installing with Manifests

You can deploy kube-state-metrics by applying Kubernetes manifests. All resources below go in the kube-system namespace. You need: ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, and Service.

1. Service Account

service-account.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
automountServiceAccountToken: false

1

kubectl apply -f service-account.yaml

2. ClusterRole

cluster-role.yaml: (grants list/watch on the API resources kube-state-metrics needs)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
rules:
  - apiGroups: [""]
    resources:
      - configmaps
      - secrets
      - nodes
      - pods
      - services
      - serviceaccounts
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumeclaims
      - persistentvolumes
      - namespaces
      - endpoints
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets", "daemonsets", "deployments", "replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["cronjobs", "jobs"]
    verbs: ["list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["list", "watch"]
  - apiGroups: ["authentication.k8s.io"]
    resources: ["tokenreviews"]
    verbs: ["create"]
  - apiGroups: ["authorization.k8s.io"]
    resources: ["subjectaccessreviews"]
    verbs: ["create"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["list", "watch"]
  - apiGroups: ["certificates.k8s.io"]
    resources: ["certificatesigningrequests"]
    verbs: ["list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses", "volumeattachments"]
    verbs: ["list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources:
      ["mutatingwebhookconfigurations", "validatingwebhookconfigurations"]
    verbs: ["list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies", "ingressclasses", "ingresses"]
    verbs: ["list", "watch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["list", "watch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings", "clusterroles", "rolebindings", "roles"]
    verbs: ["list", "watch"]

1

kubectl apply -f cluster-role.yaml

3. ClusterRoleBinding

cluster-role-binding.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: kube-system

1

kubectl apply -f cluster-role-binding.yaml

4. Deployment

deployment.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 2.8.2
    spec:
      serviceAccountName: kube-state-metrics
      automountServiceAccountToken: true
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: kube-state-metrics
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.2
          ports:
            - name: http-metrics
              containerPort: 8080
            - name: telemetry
              containerPort: 8081
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /
              port: 8081
            initialDelaySeconds: 5
            timeoutSeconds: 5
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
            runAsUser: 65534

1

kubectl apply -f deployment.yaml

5. Service

service.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
spec:
  clusterIP: None
  ports:
    - name: http-metrics
      port: 8080
      targetPort: http-metrics
    - name: telemetry
      port: 8081
      targetPort: telemetry
  selector:
    app.kubernetes.io/name: kube-state-metrics

1

kubectl apply -f service.yaml

Installing with Helm

You can install kube-state-metrics using the prometheus-community Helm chart.

1
2
3

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics -n kube-system --create-namespace

To override settings (e.g. resources, replicas, metric allowlist), create a values.yaml and pass it to helm install or helm upgrade:

1
2

helm show values prometheus-community/kube-state-metrics
helm install -f values.yaml kube-state-metrics prometheus-community/kube-state-metrics -n kube-system

The chart installs the Deployment and Service in the namespace you specify (e.g. kube-system). Replace values.yaml with the path to your overrides file.

Verifying the Installation

  1. Check that the pod is running:

    1

    kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics

  2. Port-forward and hit the metrics endpoint:

    1

    kubectl port-forward -n kube-system svc/kube-state-metrics 8080:8080

    In another terminal:

    1

    curl http://127.0.0.1:8080/metrics

    You should see Prometheus-format metrics (e.g. kube_node_info, kube_pod_status_phase).

  3. Filter a specific metric (example):

    1

    curl -s http://127.0.0.1:8080/metrics | grep kube_node_status_capacity

Configuring Prometheus to Scrape kube-state-metrics

Add a scrape job to your Prometheus configuration so Prometheus collects these metrics. If kube-state-metrics is in kube-system and the Service name is kube-state-metrics:

1
2
3
4

scrape_configs:
  - job_name: "kube-state-metrics"
    static_configs:
      - targets: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]

Reload or restart Prometheus after changing the config. Then you can query metrics (e.g. in Prometheus UI or Grafana). For a full Prometheus-on-Kubernetes setup, see How to Setup Prometheus Monitoring on Kubernetes.

Scaling and Resource Tuning

As the cluster grows, the /metrics response can become larger and slower. Allocate enough CPU and memory so kube-state-metrics stays responsive.

Suggested starting values: around 250Mi memory and 0.1 cores (100m CPU). Increase as needed. Setting CPU too low can cause high latency or OOMs.

Example override with Helm (e.g. in resources.yaml):

1
2
3
4
5
6
7

resources:
  limits:
    cpu: 500m
    memory: 768Mi
  requests:
    cpu: 200m
    memory: 512Mi

Apply with:

1

helm upgrade --reuse-values -f resources.yaml kube-state-metrics prometheus-community/kube-state-metrics -n kube-system

(Remove the trailing . from the chart name if you had it; the correct chart reference is prometheus-community/kube-state-metrics.)

Limiting Exposed Metrics

To reduce cardinality and scrape size, you can expose only a subset of metrics using the chart’s metric allowlist (or the container’s --metric-allowlist if running from manifests).

Example values.yaml for Helm:

1
2
3
4

metricAllowlist:
  - kube_node_info
  - kube_node_status_capacity
  - kube_job_status_active

Then:

1

helm upgrade --reuse-values -f values.yaml kube-state-metrics prometheus-community/kube-state-metrics -n kube-system

kube-state-metrics will then expose only the metrics in the allowlist. See the chart values and kube-state-metrics docs for options.

Summary

  • kube-state-metrics exposes Kubernetes object state metrics (pods, nodes, deployments, etc.) in Prometheus format on an HTTP endpoint. It does not report resource usage; use the metrics server for that.
  • Install via manifests (ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, Service) in kube-system or via the Helm chart prometheus-community/kube-state-metrics.
  • Verify with kubectl get pods, port-forward, and curl .../metrics. Add a Prometheus scrape job for kube-state-metrics.kube-system.svc.cluster.local:8080.
  • Scale by increasing CPU/memory; use metric allowlisting to reduce cardinality.

To build full dashboards and alerts, combine kube-state-metrics with Prometheus and Grafana. For Prometheus on Kubernetes, see How to Setup Prometheus Monitoring on Kubernetes and Monitor Linux Server With Prometheus and Grafana.

comments powered by Disqus
© 2020 - 2026 Citizix, Ltd.
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy