User Tools

Site Tools



The order of things

Create rules

sudo vim /etc/prometheus/uptime_rule.uml
..add your rule code here...

Blackbox Exporter

Blackbox Exporter by Prometheus allows probing over endpoints such as http, https, icmp, tcp and dns

Install the Blackbox Exporter

sudo useradd --no-create-home --shell /bin/false blackbox_exporter
aunpack ./blackbox_exporter-0.19.0.linux-amd64.tar.gz
sudo cp blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
rm -rf blackbox_exporter-0.19.0.linux-amd64*
# Config
sudo mkdir /etc/blackbox_exporter
sudo vim /etc/blackbox_exporter/blackbox.yml

    prober: http
    timeout: 5s
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4" # used for "tcp", defaults to "ip4"

sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
sudo vim /etc/systemd/system/blackbox_exporter.service

Description=Blackbox Exporter

ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml


sudo systemctl daemon-reload
sudo systemctl start blackbox_exporter
sudo systemctl enable blackbox_exporter

Configure Prometheus

Next, we need to provide context to prometheus on what to monitor. We will inform prometheus to monitor a web endpoint on port 8080 using the blackbox exporter (we will create a python simplehttpserver to run on port 8080). Edit the prometheus config /etc/prometheus/prometheus.yml and append the following:

sudo vim /etc/prometheus/prometheus.yml

  - job_name: 'blackbox'
    metrics_path: /probe
      module: [http_2xx]
      - targets:
        - http://localhost:8080
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Open a new terminal, create a index.html:

sudo pip3 install simple_http_server
echo "ok" > index.html
python -m SimpleHTTPServer 8080 or
python3 -m http.server 8080

Head back to the previous terminal session and restart prometheus:

sudo systemctl restart prometheus

Configure the Alarm definition:

sudo vim /etc/prometheus/blackbox_rule.yml

- name: alert.rules
  - alert: EndpointDown
    expr: probe_success == 0
    for: 10s
      severity: "critical"
      summary: "Endpoint  down"

sudo chown prometheus:prometheus /etc/prometheus/blackbox_rule.yml

promtool check rules /etc/prometheus/blackbox_rule.yml 
sudo systemctl restart prometheus

Setup Alert Manager

# Create the user for alertmanager
useradd --no-create-home --shell /bin/false alertmanager
# Download alertmanager and extract
aunpack ./alertmanager-0.23.0.linux-amd64.tar.gz 
# Move alertmanager and amtool birnaries in place
sudo cp ./alertmanager-0.23.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp ./alertmanager-0.23.0.linux-amd64/amtool /usr/local/bin/
# Ensure that the correct permissions are in place
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
# Cleanup
rm -rf alertmanager-0.23.0.linux-amd64*
# Configure Alertmanager
## Create the alertmanager directory and configure the global alertmanager configuration
sudo mkdir /etc/alertmanager
vim /etc/alertmanager/alertmanager.yml
# Ensure the permissions are in place
chown alertmanager:alertmanager -R /etc/alertmanager
# Create the alertmanager systemd unit file
sudo vim /etc/systemd/system/alertmanager.service
# Now we need to inform prometheus that we will send alerts to alertmanager to it’s exposed port
sudo vim /etc/prometheus/prometheus.yml
sudo vim /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload
sudo systemctl restart alertmanager.service 
sudo systemctl restart prometheus.service 
sudo systemctl status prometheus.service
sudo systemctl status alertmanager.service 
## Access alertmanager on your endpoint on port 9093

To Debug:

sudo journalctl -u alertmanager -f

Telegram Bot for Alertmanager
sudo pip3 install -r ./requirements.txt
sudo pip3 install python-dateutil



To get ping [icmp] to work, I had to do the following:

sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter 
prometheus.txt · Last modified: 2021/10/18 03:47 by vissie