Prometheus Alertmanager receives alerts from the Prometheus server, deduplicates and groups them, and routes notifications to receivers such as Slack, email, PagerDuty, or OpsGenie. It also handles silencing and inhibition. Once Prometheus is scraping targets and evaluating rules, it can send alerts to Alertmanager when metrics cross thresholds—for example when a target is down. This guide walks through installing and configuring Alertmanager on Linux with systemd, wiring Prometheus to Alertmanager, creating an alert rule, and sending Slack notifications when a target is unavailable.

In this guide you’ll learn:

• How to install Prometheus Alertmanager on Linux and run it with systemd
• How to configure Prometheus to use Alertmanager and scrape its metrics
• How to create a Prometheus alert rule (e.g. InstanceDown when up == 0)
• How to configure Alertmanager with a Slack receiver so you get notifications in Slack

Prerequisites

You need a working Prometheus setup before proceeding. If you need to install or configure Prometheus first, see these guides:

• How to set up Prometheus Node Exporter in Kubernetes
• How to install and configure Prometheus on a Linux server
• How to monitor Linux servers using Prometheus Node Exporter
• How to run Prometheus with Docker and Docker Compose

Installing Alertmanager

Alertmanager is distributed as a tarball on the Prometheus downloads page. Download the latest Linux release. Example (replace the version if needed):

1
2
3
4

$ curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz

$ ls
alertmanager-0.25.0.linux-amd64.tar.gz

Once the download is complete, extract it and move to the /opt/alertmanager directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ tar -xzvf alertmanager-0.25.0.linux-amd64.tar.gz

alertmanager-0.25.0.linux-amd64/
alertmanager-0.25.0.linux-amd64/alertmanager.yml
alertmanager-0.25.0.linux-amd64/NOTICE
alertmanager-0.25.0.linux-amd64/amtool
alertmanager-0.25.0.linux-amd64/alertmanager
alertmanager-0.25.0.linux-amd64/LICENSE

$ sudo mv -v alertmanager-0.25.0.linux-amd64 /opt/alertmanager

The directory contains the alertmanager binary and the alertmanager.yml configuration file. Since we’re running Alertmanager as the prometheus user (created during your Prometheus setup), give that user ownership of the directory:

1

sudo chown -Rfv prometheus:prometheus /opt/alertmanager

Creating a Data Directory

Alertmanager needs a data directory for its storage. Because the service runs as the prometheus system user, that user must own the directory (read, write, and execute).

You can create the data/ directory in the /opt/alertmanager/ directory as follows:

1
2

sudo mkdir -v /opt/alertmanager/data
sudo chown -Rfv prometheus:prometheus /opt/alertmanager/data

Create a systemd unit for Alertmanager

Use systemd to manage Alertmanager so you can start, stop, restart, and enable it on boot. Create the unit file:

1

sudo vim /etc/systemd/system/alertmanager.service

And add the following content to the file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

[Unit]
Description=Alertmanager for prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/alertmanager/alertmanager \
  --config.file=/opt/alertmanager/alertmanager.yml \
  --storage.path=/opt/alertmanager/data
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

Save and exit the file.

For the systemd changes to take effect, run the following command:

1

sudo systemctl daemon-reload

Now, start the alertmanager service with the following command:

1

sudo systemctl start alertmanager

Confirm that the service is running as expected by checking its status:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ sudo systemctl status alertmanager
● alertmanager.service - Alertmanager for prometheus
   Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-02-09 10:02:37 UTC; 2h 27min ago
 Main PID: 599774 (alertmanager)
    Tasks: 8 (limit: 23406)
   Memory: 27.8M
   CGroup: /system.slice/alertmanager.service
           └─599774 /opt/alertmanager/alertmanager --config.file=/opt/alertmanager/alertmanager.yml --storage.path=/opt/alertmanager/data

Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.309Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.309Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)"
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.327Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=10.18.0.10 port=9094
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.340Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.382Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/opt/alertmanager/alertmanager.yml
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.383Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/opt/alertmanager/alertmanager.yml
Feb 09 10:02:37 monitoring alertmanager[599774]: ts=2023-02-09T10:02:37.388Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093

Add the alertmanager service to the system startup so that it automatically starts on boot with the following command:

1

sudo systemctl enable alertmanager

Configuring Prometheus

Next, configure Prometheus to use Alertmanager and (optionally) scrape Alertmanager’s metrics. Both are done in this section.

Add a scrape job for Alertmanager in the scrape_configs section of prometheus.yml. If Alertmanager and Prometheus run on the same host, use 127.0.0.1:9093; otherwise use your Alertmanager host IP. You can find the host IP using this command:

1

hostname -I

Example prometheus.yml with an Alertmanager scrape job:

1
2
3
4
5
6
7

...
# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  - job_name: 'alertmanager'
    static_configs:
    - targets: ['127.0.0.1:9093']
...

Also add Alertmanager as the alert receiver in the alerting.alertmanagers section of prometheus.yml:

1
2
3
4
5
6
7
8

...
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 127.0.0.1:9093
...

For the changes to take effect, restart the prometheus service as follows:

1

sudo systemctl restart prometheus

Open http://<server_ip>:9090/targets in your browser; the alertmanager target should be UP, so Prometheus is successfully talking to Alertmanager.

Creating a Prometheus Alert Rule

On Prometheus, you can use the up expression to find the state of the targets added to Prometheus in the graph search section.

Targets in the UP state (reachable by Prometheus) have the value 1; targets that are DOWN (not running or unreachable) have the value 0.

If you stop one of the targets node_exporter (let’s say).

1

sudo systemctl stop node_exporter

The UP value of that target in prometheus should be 0. So, you can use the up == 0 expressions to list only the targets that are not running or inaccessible to Prometheus.

You can use this expression in a Prometheus alert rule so that when one or more targets are down, Prometheus sends the alert to Alertmanager.

To create a Prometheus Alert, create a new file rules.yml in the /opt/prometheus/ directory as follows:

1

sudo vim /opt/prometheus/rules.yml

Now, type in the following lines in the rules.yml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

groups:
- name: Instances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    # Labels - additional labels to be attached to the alert
    labels:
      severity: 'critical'
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      summary: 'Instance {{ $labels.instance }} down'
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'

Here, the alert InstanceDown will be fired when targets are not running or inaccessible to Prometheus (that is up == 0) for a minute (1m).

Now, update the Prometheus configuration file /opt/prometheus/prometheus.yml as follows:

1
2
3
4
5

...
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/opt/prometheus/rules.yml"
...

Another important option of the prometheus.yml file is evaluation_interval. Prometheus will check whether any rules matched every evaluation_interval time. The default is 15s (15 seconds). So, the Alert rules in the rules.yml file will be checked every 15 seconds.

For the changes to take effect, restart the prometheus service:

1

sudo systemctl restart prometheus

Now, navigate to the URL http://<server_ip>:9090/rules from your favorite web browser, and you should see the rule InstanceDown that you’ve just added.

Navigate to the URL http://<server_ip>:9090/alerts from your favorite web browser, and you should see the state of the alert InstanceDown.

Because you stopped node_exporter earlier, the alert is active and will be sent to Alertmanager once it has been firing for 1 minute.

After a minute, the InstanceDown alert should show as FIRING and Prometheus will have sent it to Alertmanager.

Configuring the Slack receiver in Alertmanager

Here we configure Slack as an Alertmanager receiver so you get messages in Slack when a Prometheus target is DOWN (or when any other configured alert fires).

If you want to receive notifications via Slack, you should be part of a Slack workspace. If you are currently not a part of any Slack workspace, or you want to test this out in separate workspace, you can quickly create one here.

To set up alerting in your Slack workspace, you’re going to need a Slack API URL. Go to Slack -> Administration -> Manage apps.

In the Manage apps directory, search for Incoming Webhooks and add it to your Slack workspace.

Next, specify in which channel you’d like to receive notifications from Alertmanager. (I’ve created #citizix-alerts channel.) After you confirm and add Incoming WebHooks integration, webhook URL (which is your Slack API URL) is displayed. Copy it.

Then edit /opt/alertmanager/alertmanager.yml. Use the webhook URL you copied as slack_api_url, and set channel to your Slack channel (e.g. #alerts):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'slack-notifications'
receivers:
  - name: 'slack-notifications'
    slack_configs:
    - channel: '#your-alerts-channel'
      send_resolved: true
      text: 'Alert group {{ .GroupLabels.alertname }} triggered'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

In the above configs, we have updated the alertmanager receiver to slack-notifications, the receiver we have created and added configs. It will use that from now on.

The repeat_interval in the route (default 1h) controls how often Alertmanager will re-send the same group of alerts. After sending a notification, it waits this long before sending again for that group. Increase it if you want fewer repeated notifications.

Now, restart the alertmanager service for the changes to take effect:

1

sudo systemctl restart alertmanager

You should get a message on Slack shortly (the InstanceDown alert for the stopped node_exporter).

Frequently Asked Questions (FAQ)

What is Prometheus Alertmanager?

Alertmanager is the component that receives alerts from Prometheus, deduplicates and groups them, and routes them to receivers such as Slack, email, or PagerDuty. It also handles silencing and inhibition so you can reduce noise and avoid duplicate notifications.

What port does Alertmanager use?

Alertmanager listens on port 9093 by default. Prometheus sends alerts to this port and can also scrape Alertmanager metrics from it for monitoring.

How do I send Prometheus alerts to Slack?

Configure a Slack Incoming Webhook in your workspace, then set slack_api_url in the global section of alertmanager.yml and add a receiver with slack_configs (including your channel). Point the route to that receiver so firing alerts are sent to Slack.

Why is my Prometheus alert not firing?

Ensure the alert rule expression (e.g. up == 0) is true for at least the for duration (e.g. 1m). Check http://<server>:9090/alerts for state (Pending vs Firing). Confirm alerting.alertmanagers in prometheus.yml points to your Alertmanager (e.g. 127.0.0.1:9093) and that rule_files includes your rules file.

Conclusion

You now have Prometheus Alertmanager installed on Linux, managed by systemd, with Prometheus configured to evaluate alert rules and send alerts to Alertmanager. By adding a Slack receiver, you get notifications in Slack when targets go down or other conditions fire. For richer alerting and enrichment (e.g. with Kubernetes), see how to set up Robusta for Kubernetes monitoring on EKS or production-ready Prometheus on Kubernetes.