How To Install and Configure Prometheus On a Linux Server

Prometheus is a free open source software application used for event monitoring and alerting. It was originally built at SoundCloud. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Metrics are numeric measurements, time series mean that changes are recorded over time. What users want to measure differs from application to application. For a web server it might be request times, for a database it might be number of active connections or number of active queries etc.

Metrics play an important role in understanding why your application is working in a certain way. If you are running a web application and find that the application is slow. You will need some information to find out what is happening with your application. For example the application can become slow when the number of requests are high. If you have the request count metric you can spot the reason and increase the number of servers to handle the load.

Components

The Prometheus ecosystem consists of multiple components, many of which are optional:

the main Prometheus server which scrapes and stores time series data
client libraries for instrumenting application code
a push gateway for supporting short-lived jobs
special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
an alertmanager to handle alerts
various support tools

Most Prometheus components are written in Go, making them easy to build and deploy as static binaries.

Step 1. Downloading Prometheus Binary

Download the binary using curl. We are using version v2.32.1 as it is the latest as of the writing of this guide.

cd /tmp
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.32.1/prometheus-2.32.1.linux-amd64.tar.gz

Step 2. Create Prometheus user

Let us create a dedicated prometheus user. This is so we can run the prometheus service as the dedicated user. Use this command:

sudo useradd --no-create-home --shell /bin/false prometheus

Step 3. Extract the binaries to executable path

Next, extract the downloaded file and move the binaries to executable path. We are moving the prometheus and promtool in the commands below:

tar -xzf prometheus-2.32.1.linux-amd64.tar.gz
sudo mv prometheus-2.32.1.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.32.1.linux-amd64/promtool /usr/local/bin/

Ensure prometheus user owns the binaries using linux chown command:

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

3. Create required directories and copy configs

Let us create the required directories that prometheus will store its config and data. Use these commands:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Move the consoles and console_libraries directories from prometheus-files to /etc/prometheus folder and change the ownership to prometheus user.

sudo mv prometheus-2.32.1.linux-amd64/consoles /etc/prometheus
sudo mv prometheus-2.32.1.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus

Step 4. Setup Prometheus Configuration

All the prometheus configurations should be present in /etc/prometheus/prometheus.yml file. Open the file:

sudo vim <span style="font-size: calc(11px + 0.2em);">/etc/prometheus/prometheus.yml</span>

Add this configuration to the file:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

Change the ownership of the file to prometheus user.

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Step 5. Create Prometheus Systemd Service File

Create a prometheus systemd service file. We will use this service to manage the service starting and stoping.

sudo vim /etc/systemd/system/prometheus.service

Copy the following content to the file.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload the systemd service to register the prometheus service and start the prometheus service.

sudo systemctl daemon-reload
sudo systemctl start prometheus

Check the prometheus service status using the following command.

$ sudo systemctl status prometheus
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-02-08 20:42:39 UTC; 8min ago
 Main PID: 234972 (prometheus)
    Tasks: 7 (limit: 23167)
   Memory: 29.4M
   CGroup: /system.slice/prometheus.service
           └─234972 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/promethe>

Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.985Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks rep>
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.985Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a whi>
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.987Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.987Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=0 maxS>
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.987Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_r>
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.993Z caller=main.go:945 level=info fs_type=XFS_SUPER_MAGIC
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.994Z caller=main.go:948 level=info msg="TSDB started"
Feb 08 20:42:39 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:39.994Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/promet>
Feb 08 20:42:40 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:40.046Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filenam>
Feb 08 20:42:40 rockysrv.citizix.com prometheus[234972]: ts=2022-02-08T20:42:40.047Z caller=main.go:897 level=info msg="Server is ready to receive web requests."

The service status should show Active(running) as shown above.

Step 6: Access Prometheus Web UI

Now you will be able to access the prometheus UI on 9090 port of the prometheus server.

http://<prometheus-server-ip>:9090/graph

If everything went as expected, you should be able to see the Prometheus UI.

You can use the prometheus query tab to query the available metrics.

Right now, we have just configured the Prometheus server. You need to register the target in the prometheus.yml file to get the metrics from the source systems.

For example, if you want to monitor ten servers, the IP address of these servers should be added as a target in the Prometheus configuration to scrape the metrics.

The server should have Node Exporter installed to collect all the system metrics and make it available for Prometheus to scrap it.