How to Set Up Prometheus with Grafana-Style Alerting via Alertmanager

Why Prometheus alone never pages you

Prometheus loads your alert rules and evaluates them every evaluation_interval, but a firing rule only becomes a row in Prometheus's own /api/v1/alerts. Prometheus does not group, deduplicate, or route notifications - it fans every firing alert out over HTTP as POST /api/v2/alerts to each Alertmanager it is told about. If you never configure an Alertmanager, alerts fire silently and no one is paged.

So the setup has two moving parts: tell Prometheus where Alertmanager is, and stand up an Alertmanager that knows what to do with the alerts it receives. This is the same Prometheus + alerting stack you pair with Grafana for dashboards - Grafana reads Prometheus, Alertmanager routes the pages.

Step 1 - add the alerting block to prometheus.yaml

alerting: is a top-level key, a peer of scrape_configs: and rule_files: - not nested under them. The minimum is a static target pointing at Alertmanager:

global:
  scrape_interval: 2s
  evaluation_interval: 5s

rule_files:
  - alerts.yaml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'api'
    static_configs:
      - targets: ['localhost:8000']

Nesting alerting: inside scrape_configs: is the most common mistake and Prometheus will simply ignore it - /api/v1/alertmanagers will show zero active peers.

Step 2 - write a minimal alertmanager.yaml

An Alertmanager config is a route: (how alerts flow) plus receivers: (where they go). One webhook receiver is enough to prove the pipe:

route:
  receiver: webhook
  group_wait: 1s
  group_interval: 1s
  repeat_interval: 1m

receivers:
  - name: webhook
    webhook_configs:
      - url: http://localhost:5001/sink
        send_resolved: false

Validate it before starting anything - amtool catches indentation and schema errors:

amtool check-config alertmanager.yaml

The webhook does not have to actually accept the POST. Alertmanager tries, fails, retries in the background, and the alert still shows as active in its API - which is exactly what you verify at the end.

Step 3 - run both binaries side by side

alertmanager --config.file=alertmanager.yaml --storage.path=/tmp/am --web.listen-address=:9093 &
prometheus  --config.file=prometheus.yaml  --storage.tsdb.path=/tmp/prom --web.listen-address=:9090 &

Confirm Prometheus discovered the Alertmanager - activeAlertmanagers should be 1:

curl http://localhost:9090/api/v1/alertmanagers | python3 -m json.tool

Step 4 - drive a synthetic failure and verify

The demo app exposes a /flag/fail toggle and an /api/maybe-fail endpoint that starts returning 500s once flipped. The HighErrorRate rule fires when the 5xx ratio crosses 1% for 5 seconds:

curl -X POST http://localhost:8000/flag/fail
for i in {1..30}; do curl -s http://localhost:8000/api/maybe-fail >/dev/null; done

Give Prometheus a couple of evaluation cycles, then confirm the alert reached Alertmanager end to end:

curl http://localhost:9093/api/v2/alerts | python3 -m json.tool

You should see a HighErrorRate entry with status.state active - proof the full path fired, forwarded, and landed.