Containers Kubernetes Monitoring

How To Setup Kube State Metrics on Kubernetes

In this article, we will learn what kube-state-metrics is, the importance of enabling kube-state-metrics in your cluster, some of its use cases, and how to implement it in your cluster.

Introduction

kube-state-metrics

kube-state-metrics is a monitoring tool that leverages the Kubernetes API to provide metrics on the state of the objects in your cluster such as the pod status, node capacity, available deployment replicas, resources requests and limits, and many more.
It is a lightweight utility that provides information about the state of a couple of Kubernetes objects by listening to Kubernetes API.

The Kubernetes metrics server is different from kube-state-metrics. The Kubernetes metrics server provides information about the usage of the cluster resources (such as CPU and memory) which is useful for scaling, while kube-state-metrics focuses more on the health of the Kubernetes objects in your cluster, such as the availability of pods and the readiness of nodes.

Following are some of the important metrics you can get from Kube state metrics.

  • Node status, node capacity (CPU and memory)
  • Replica-set compliance (desired/available/unavailable/updated status of replicas per deployment)
  • Pod status (waiting, running, ready, etc)
  • Ingress metrics
  • PV, PVC metrics
  • Daemonset & Statefulset metrics.
    Resource requests and limits.
  • Job & Cronjob metrics

You can check out the detailed supported metrics from the documentation here.

kube-state-metrics benefits

kube-state-metrics has these benefits:

  • Availability of metrics data at an HTTP endpoint
  • Data available in Prometheus exposition format
  • Reports raw data from Kube API

Availability of Metrics Data at a HTTP Endpoint

kube-state-metrics exposes its data on the /metrics HTTP endpoint and is served as plain text, which makes it easy to configure it with any monitoring platform that scrapes data from the Kubernetes cluster. you can also view kube-state-metrics in a web browser, though the data displayed is in a raw format that requires additional tools like Prometheus and Grafana to become more human-readable.

Prometheus Exposition Format

Prometheus is a popular tool that scrapes metrics from its target endpoint and saves the scraped data in an internal time-series database for analysis. This can be integrated with another monitoring tool such as Grafana to visualize the data.

Explaining Prometheus is out of the scope of this article. In summary, these are some few key points about Prometheus:

  • Metric Collection: Prometheus uses the pull model to retrieve metrics over HTTP. There is an option to push metrics to Prometheus using Pushgateway for use cases where Prometheus cannot Scrape the metrics. One such example is collecting custom metrics from short-lived kubernetes jobs & Cronjobs
  • Metric Endpoint: The systems that you want to monitor using Prometheus should expose the metrics on an /metrics endpoint. Prometheus uses this endpoint to pull the metrics in regular intervals.
  • PromQL: Prometheus comes with PromQL, a very flexible query language that can be used to query the metrics in the Prometheus dashboard. Also, the PromQL query will be used by Prometheus UI and Grafana to visualize metrics.
  • Prometheus Exporters: Exporters are libraries that convert existing metrics from third-party apps to Prometheus metrics format. There are many official and community Prometheus exporters. One example is, the Prometheus node exporter. It exposes all Linux system-level metrics in Prometheus format.
  • TSDB (time-series database): Prometheus uses TSDB for storing all the data efficiently. By default, all the data gets stored locally. However, to avoid a single point of failure, there are options to integrate remote storage for Prometheus TSDB.

The output of kube-state-metrics uses the Prometheus exposition format, which exposes your metrics data in such a way that Prometheus can understand and use. This makes integrating kube-state-metrics with Prometheus straightforward.

Implementing kube-state-metrics

Installing using Manifest

Kube state metrics is available as a public docker image. You will have to deploy the following Kubernetes objects for Kube state metrics to work.

  • A Service Account
  • Cluster Role – For kube state metrics to access all the Kubernetes API objects.
  • Cluster Role Binding – Binds the service account with the cluster role.
  • Kube State Metrics Deployment
  • Service – To expose the metrics

All the above Kube state metrics objects will be deployed in the kube-system namespace

Service Account

Save the following as service-account.yaml:

apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
  name: kube-state-metrics
  namespace: kube-system

Then create with this command:

kubectl apply -f service-account.yaml

Cluster Role

Save this as cluster-role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - serviceaccounts
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  - ingressclasses
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  - rolebindings
  - roles
  verbs:
  - list
  - watch

Create with this command:

kubectl apply -f cluster-role.yaml

ClusterRoleBinding

Save the following as cluster-role-binding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

Then create with the following command:

kubectl create -f cluster-role-binding.yaml

Deployment

Save the following as deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 2.8.2
    spec:
      automountServiceAccountToken: true
      containers:
      - image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.2
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http-metrics
        - containerPort: 8081
          name: telemetry
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsUser: 65534
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: kube-state-metrics

Then create with the following command:

kubectl create -f deployment.yaml

Service

Save the following as service.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.2
  name: kube-state-metrics
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
  - name: telemetry
    port: 8081
    targetPort: telemetry
  selector:
    app.kubernetes.io/name: kube-state-metrics

Then create with the following command:

kubectl create -f service.yaml

Installing using Helm

You can also use Helm to install kube-state-metrics. Add the prometheus-community helm chart repository using this command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Then update helm repository using the following command:

helm repo update

Finally, install kube-state-metrics using the following command:

helm install kube-state-metrics prometheus-community/kube-state-metrics

You can override the default configurations of kube-state-metrics, such as the metrics it exposes, memory limit, and number of replicas. To view the available configurations, run the following command.

helm show values prometheus-community/kube-state-metrics

Create a values.yaml file that includes the values you want to override, and run the command below.

helm install -f values.yaml kube-state-metrics prometheus-community/kube-state-metrics

This installs kube-state-metrics with the configuration values that you specified in the config.yaml file.

kube-state-metrics has been successfully installed in your cluster. Now you have to expose it, so it will be available at your localhost endpoint. Run the command below:

kubectl port-forward -n kube-system svc/kube-state-metrics 8080:8080

You can test by sending curl in terminal:

curl http://127.0.0.1:8080/metrics

The response contains a lot of information, which can make it difficult to find the exact metric you want. You can use a combination of grep and curl requests to surface the metric you want. The example below will return the node status capacity.

curl 127.0.0.1:8080/metrics | grep kube_node_status_capacity

Scaling Considerations for kube-state-metrics

you might experience increased latency on the /metrics endpoint as your cluster grows.

begin by allocating 250Mib of memory and 0.1 cores for kube-state-metrics, which you can increase as needed as your cluster grows in size. It’s important to note that setting the CPU value too low when configuring kube-state-metrics can affect its performance. If you find that you need to allocate more resources, you can do so by creating a resources.yaml file as follows.

resources:
  limits
  - cpu: 500m
  - memory: 768Mi
  requests:
  - cpu: 200m
  - memory: 512Mi

Then pass it to helm upgrade:

helm upgrade --reuse-values -f resources.yaml kube-state-metrics prometheus-community/kube-state-metrics.

Exposing specific metrics

kube-state-metrics allows you to only expose the metrics that you need.
To do this, include the following content in the values.yaml file:

# values.yaml
metricAllowlist:
 - kube_node_info
 - kube_job_status_active

Then run:

helm upgrade --reuse-values -f values.yaml kube-state-metrics prometheus-community/kube-state-metrics

Once this is done, kube-state-metrics will only expose kube_node_info and kube_job_status_active metrics as configured above.

Kube State Metrics Prometheus Config

All the Kube static metrics can be obtained from the Kube state service endpoint on /metrics URI.

This configuration can be added as part of the Prometheus job configuration. You need to add the following job configuration to your Prometheus config for Prometheus to scrape all the Kube state metrics.

- job_name: 'kube-state-metrics'
  static_configs:
    - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

Conclusion

In this article, we learnt all about kube-state-metrics, how to set it up in kubernetes and its benefits.
In as much as kube-state-metrics is a powerful tool for monitoring your cluster, you still need to integrate it with other monitoring tools like Prometheus and Grafana or the Kubernetes dashboard to unleash its full capabilities.

comments powered by Disqus
© 2020 - 2024 Citizix, Inc.
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy