Differences

This shows you the differences between two versions of the page.

--- prometheus [2021/10/11 05:33] – created vissie
+++ prometheus [2021/10/18 03:47] (current) – vissie
@@ Line 1: / Line 1: @@
+=====The order of things=====
+  * https://blog.ruanbekker.com/blog/2019/05/07/setup-prometheus-and-node-exporter-on-ubuntu-for-epic-monitoring/
+  * https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
+  * https://www.scaleway.com/en/docs/tutorials/prometheus-monitoring-grafana-dashboard/
+  - Install/Configure Prometheus
+  - Install/Configure Node Exporter
+  - Install/Configure Nginx Reverse Proxy
+  - Install/Configure Alert Manager
+  - Install/Configure Blackbox Exporter
+=====Create rules=====
+  sudo vim /etc/prometheus/uptime_rule.uml
+  ..add your rule code here...
+=====Blackbox Exporter=====
+Blackbox Exporter by Prometheus allows probing over endpoints such as http, https, icmp, tcp and dns
+====Install the Blackbox Exporter====
+https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
+  sudo useradd --no-create-home --shell /bin/false blackbox_exporter
+  wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
+  aunpack ./blackbox_exporter-0.19.0.linux-amd64.tar.gz
+  sudo cp blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
+  sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
+  rm -rf blackbox_exporter-0.19.0.linux-amd64*
+  # Config
+  sudo mkdir /etc/blackbox_exporter
+  sudo vim /etc/blackbox_exporter/blackbox.yml
+<sxh bash; gutter: false>
+modules:
+  http_2xx:
+    prober: http
+    timeout: 5s
+    http:
+      valid_status_codes: []
+      method: GET
+      preferred_ip_protocol: "ip4" # used for "tcp", defaults to "ip4"
+</sxh>
+  sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
+  sudo vim /etc/systemd/system/blackbox_exporter.service
+<sxh bash; gutter: false>
+[Unit]
+Description=Blackbox Exporter
+Wants=network-online.target
+After=network-online.target
+[Service]
+User=blackbox_exporter
+Group=blackbox_exporter
+Type=simple
+ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml
+[Install]
+WantedBy=multi-user.target
+</sxh>
+  sudo systemctl daemon-reload
+  sudo systemctl start blackbox_exporter
+  sudo systemctl enable blackbox_exporter
+====Configure Prometheus====
+Next, we need to provide context to prometheus on what to monitor. We will inform prometheus to monitor a web endpoint on port 8080 using the blackbox exporter (we will create a python simplehttpserver to run on port 8080).
+Edit the prometheus config /etc/prometheus/prometheus.yml and append the following:
+  sudo vim /etc/prometheus/prometheus.yml
+<sxh bash; gutter: false>
+  - job_name: 'blackbox'
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
+    static_configs:
+      - targets:
+        - http://localhost:8080
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: localhost:9115
+</sxh>
+Open a new terminal, create a index.html:
+  sudo pip3 install simple_http_server
+  echo "ok" > index.html
+  python -m SimpleHTTPServer 8080 or
+  python3 -m http.server 8080
+Head back to the previous terminal session and restart prometheus:
+  sudo systemctl restart prometheus
+====Configure the Alarm definition:====
+  sudo vim /etc/prometheus/blackbox_rule.yml
+<sxh bash; gutter: false>
+groups:
+- name: alert.rules
+  rules:
+  - alert: EndpointDown
+    expr: probe_success == 0
+    for: 10s
+    labels:
+      severity: "critical"
+    annotations:
+      summary: "Endpoint  down"
+</sxh>
+  sudo chown prometheus:prometheus /etc/prometheus/blackbox_rule.yml
+  promtool check rules /etc/prometheus/blackbox_rule.yml
+  sudo systemctl restart prometheus
 =====Setup Alert Manager=====
   # Create the user for alertmanager
@@ Line 31: / Line 144: @@
   ## Access alertmanager on your endpoint on port 9093
+To Debug:
+  sudo journalctl -u alertmanager -f
+=====Telegram Bot for Alertmanager=====
+  https://github.com/nopp/alertmanager-webhook-telegram-python.git
+  sudo pip3 install -r ./requirements.txt
+  sudo pip3 install python-dateutil
+=====Issues=====
+====ICMP====
+To get ping [icmp] to work, I had to do the following:
+  sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter

The World According To Vissie

User Tools

Site Tools

Differences

Page Tools