The World According To Vissie

• https://blog.ruanbekker.com/blog/2019/05/07/setup-prometheus-and-node-exporter-on-ubuntu-for-epic-monitoring/
• https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
• https://www.scaleway.com/en/docs/tutorials/prometheus-monitoring-grafana-dashboard/

1. Install/Configure Prometheus
2. Install/Configure Node Exporter
3. Install/Configure Nginx Reverse Proxy
4. Install/Configure Alert Manager
5. Install/Configure Blackbox Exporter

sudo vim /etc/prometheus/uptime_rule.uml
..add your rule code here...

Blackbox Exporter by Prometheus allows probing over endpoints such as http, https, icmp, tcp and dns

https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/

sudo useradd --no-create-home --shell /bin/false blackbox_exporter
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
aunpack ./blackbox_exporter-0.19.0.linux-amd64.tar.gz
sudo cp blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
rm -rf blackbox_exporter-0.19.0.linux-amd64*
# Config
sudo mkdir /etc/blackbox_exporter
sudo vim /etc/blackbox_exporter/blackbox.yml

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4" # used for "tcp", defaults to "ip4"

sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
sudo vim /etc/systemd/system/blackbox_exporter.service

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start blackbox_exporter
sudo systemctl enable blackbox_exporter

Next, we need to provide context to prometheus on what to monitor. We will inform prometheus to monitor a web endpoint on port 8080 using the blackbox exporter (we will create a python simplehttpserver to run on port 8080). Edit the prometheus config /etc/prometheus/prometheus.yml and append the following:

sudo vim /etc/prometheus/prometheus.yml

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://localhost:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Open a new terminal, create a index.html:

sudo pip3 install simple_http_server
echo "ok" > index.html
python -m SimpleHTTPServer 8080 or
python3 -m http.server 8080

Head back to the previous terminal session and restart prometheus:

sudo systemctl restart prometheus

sudo vim /etc/prometheus/blackbox_rule.yml

groups:
- name: alert.rules
  rules:
  - alert: EndpointDown
    expr: probe_success == 0
    for: 10s
    labels:
      severity: "critical"
    annotations:
      summary: "Endpoint  down"

sudo chown prometheus:prometheus /etc/prometheus/blackbox_rule.yml

promtool check rules /etc/prometheus/blackbox_rule.yml 
sudo systemctl restart prometheus

# Create the user for alertmanager
useradd --no-create-home --shell /bin/false alertmanager
# Download alertmanager and extract
wget https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz
aunpack ./alertmanager-0.23.0.linux-amd64.tar.gz 
# Move alertmanager and amtool birnaries in place
sudo cp ./alertmanager-0.23.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp ./alertmanager-0.23.0.linux-amd64/amtool /usr/local/bin/
# Ensure that the correct permissions are in place
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
# Cleanup
rm -rf alertmanager-0.23.0.linux-amd64*
# Configure Alertmanager
## Create the alertmanager directory and configure the global alertmanager configuration
sudo mkdir /etc/alertmanager
vim /etc/alertmanager/alertmanager.yml
# Ensure the permissions are in place
chown alertmanager:alertmanager -R /etc/alertmanager
# Create the alertmanager systemd unit file
sudo vim /etc/systemd/system/alertmanager.service
# Now we need to inform prometheus that we will send alerts to alertmanager to it’s exposed port
sudo vim /etc/prometheus/prometheus.yml
sudo vim /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload
sudo systemctl restart alertmanager.service 
sudo systemctl restart prometheus.service 
sudo systemctl status prometheus.service
sudo systemctl status alertmanager.service 
## Access alertmanager on your endpoint on port 9093

To Debug:

sudo journalctl -u alertmanager -f

https://github.com/nopp/alertmanager-webhook-telegram-python.git
sudo pip3 install -r ./requirements.txt
sudo pip3 install python-dateutil

To get ping [icmp] to work, I had to do the following:

sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter