Prometheus is a free open source software application used for event monitoring and alerting. It was originally built at SoundCloud. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.
Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Metrics are numeric measurements, time series mean that changes are recorded over time. What users want to measure differs from application to application. For a web server it might be request times, for a database it might be number of active connections or number of active queries etc.
Metrics play an important role in understanding why your application is working in a certain way. If you are running a web application and find that the application is slow. You will need some information to find out what is happening with your application. For example the application can become slow when the number of requests are high. If you have the request count metric you can spot the reason and increase the number of servers to handle the load.
The Prometheus ecosystem consists of multiple components, many of which are optional:
- the main Prometheus server which scrapes and stores time series data
- client libraries for instrumenting application code
- a push gateway for supporting short-lived jobs
- special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
- an alertmanager to handle alerts
- various support tools
Most Prometheus components are written in Go, making them easy to build and deploy as static binaries.
Step 1. Downloading Prometheus Binary
Download the binary using curl. We are using version v2.32.1 as it is the latest as of the writing of this guide.
cd /tmp curl -LO https://github.com/prometheus/prometheus/releases/download/v2.32.1/prometheus-2.32.1.linux-amd64.tar.gz
Step 2. Create Prometheus user
Let us create a dedicated prometheus user. This is so we can run the prometheus service as the dedicated user. Use this command:
sudo useradd --no-create-home --shell /bin/false prometheus
Step 3. Extract the binaries to executable path
Next, extract the downloaded file and move the binaries to executable path. We are moving the prometheus and promtool in the commands below:
tar -xzf prometheus-2.32.1.linux-amd64.tar.gz sudo mv prometheus-2.32.1.linux-amd64/prometheus /usr/local/bin/ sudo mv prometheus-2.32.1.linux-amd64/promtool /usr/local/bin/
Ensure prometheus user owns the binaries using linux
sudo chown prometheus:prometheus /usr/local/bin/prometheus sudo chown prometheus:prometheus /usr/local/bin/promtool
3. Create required directories and copy configs
Let us create the required directories that prometheus will store its config and data. Use these commands:
sudo mkdir /etc/prometheus sudo mkdir /var/lib/prometheus sudo chown prometheus:prometheus /etc/prometheus sudo chown prometheus:prometheus /var/lib/prometheus
console_libraries directories from prometheus-files to
/etc/prometheus folder and change the ownership to
sudo mv prometheus-2.32.1.linux-amd64/consoles /etc/prometheus sudo mv prometheus-2.32.1.linux-amd64/console_libraries /etc/prometheus sudo chown -R prometheus:prometheus /etc/prometheus
Step 4. Setup Prometheus Configuration
All the prometheus configurations should be present in
/etc/prometheus/prometheus.yml file. Open the file:
sudo vim /etc/prometheus/prometheus.yml
Add this configuration to the file:
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: "prometheus" # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ["localhost:9090"]
Change the ownership of the file to prometheus user.
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Step 5. Create Prometheus Systemd Service File
Create a prometheus systemd service file. We will use this service to manage the service starting and stoping.
sudo vim /etc/systemd/system/prometheus.service
Copy the following content to the file.
[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries [Install] WantedBy=multi-user.target
Reload the systemd service to register the prometheus service and start the prometheus service.
sudo systemctl daemon-reload sudo systemctl start prometheus
Check the prometheus service status using the following command.
$ sudo systemctl status prometheus ● prometheus.service - Prometheus Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2022-02-08 20:42:39 UTC; 8min ago Main PID: 234972 (prometheus) Tasks: 7 (limit: 23167) Memory: 29.4M CGroup: /system.slice/prometheus.service └─234972 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/promethe> Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.985Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks rep> Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.985Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a whi> Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.987Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.987Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=0 maxS> Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.987Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_r> Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.993Z caller=main.go:945 level=info fs_type=XFS_SUPER_MAGIC Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.994Z caller=main.go:948 level=info msg="TSDB started" Feb 08 20:42:39 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:39.994Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/promet> Feb 08 20:42:40 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:40.046Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filenam> Feb 08 20:42:40 rockysrv.citizix.com prometheus: ts=2022-02-08T20:42:40.047Z caller=main.go:897 level=info msg="Server is ready to receive web requests."
The service status should show Active(running) as shown above.
Step 6: Access Prometheus Web UI
Now you will be able to access the prometheus UI on 9090 port of the prometheus server.
If everything went as expected, you should be able to see the Prometheus UI.
You can use the prometheus query tab to query the available metrics.
Right now, we have just configured the Prometheus server. You need to register the target in the
prometheus.yml file to get the metrics from the source systems.
For example, if you want to monitor ten servers, the IP address of these servers should be added as a target in the Prometheus configuration to scrape the metrics.
The server should have Node Exporter installed to collect all the system metrics and make it available for Prometheus to scrap it.