In this guide we are going to check how to monitor linux servers with Grafana and prometheus.

Grafana is a graphing tool, it takes data and displays them in a visually appealing maner that is easy to interpret and analyze. Prometheus is a metric server, it collects metric data periodically from a specified host such as a running linux server.

Grafana

Grafana is an opensource visualization tool. It allows you to visualize tools using dashboards that take information from server monitoring tool like Prometheus then display this information.

Prometheus

Prometheus is an open source server and application metrics tool. it can also be configured to alert.
Grafana and prometheus are widely adopted and are open source thus popular for monitoring.

Installing Node Exporter

The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics.

node_exporter is an official package that should be installed on Linux servers to be monitored. It exposes multiple hardware and OS metrics, which will be pulled by Prometheus and eventually visualized on Grafana.

Get the latest release from the release page here https://github.com/prometheus/node_exporter/releases

curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.2.0/node_exporter-1.2.0.linux-amd64.tar.gz

tar -xzvf node_exporter-1.2.0.linux-amd64.tar.gz

sudo mv node_exporter-1.2.0.linux-amd64/node_exporter /usr/local/bin/

Create a systemd file /etc/systemd/system/node_exporter.service to manage the service

# Create prometheus user
sudo useradd -s /sbin/false prometheus

cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]

Description=node_exporter
Wants=network-online.target
After=network-online.target

[Service]

User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]

WantedBy=multi-user.target
EOF

Start Node Exporter

sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter

Node Exporter metrics

Once the Node Exporter is installed and running, you can verify that metrics are being exported by cURLing the /metrics endpoint:

curl http://localhost:9100/metrics

Adding Server to Prometheus

Your locally running Prometheus instance needs to be properly configured in order to access Node Exporter metrics. The following prometheus.yml example configuration file will tell the Prometheus instance to scrape, and how frequently, from the Node Exporter via localhost:9100:

global:
  scrape_interval: 15s

scrape_configs:
- job_name: node_exporter
  static_configs:
  - targets: [&#039;localhost:9100&#039;]

Create a prometheus.yml config file with jobs

cat > /etc/prometheus/prometheus.yml <<EOF
- job_name: 'node_exporter'
  basic_auth:
    username: 132668
    password: eyJrIjoiODYyMmFiNTVjNzc1ZmJhODlkOWIzNWRjYjRjZWVmZTY1NjIxZWUyOCIsIm4iOiJtYWluLXB1Ymxpc2gta2V5IiwiaWQiOjUwNzY3OX0=
  static_configs:
    - targets: ['https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push']
EOF

sudo chown -R prometheus:prometheus /etc/prometheus

Use this node exporter dashboard
https://grafana.com/grafana/dashboards/1860