The World According To Vissie

User Tools

Site Tools

Trace:
prometheus

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
prometheus [2021/10/11 05:56] – vissieprometheus [2021/10/18 03:47] (current) – vissie
Line 1: Line 1:
 +=====The order of things=====
 +  * https://blog.ruanbekker.com/blog/2019/05/07/setup-prometheus-and-node-exporter-on-ubuntu-for-epic-monitoring/
 +  * https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
 +  * https://www.scaleway.com/en/docs/tutorials/prometheus-monitoring-grafana-dashboard/
 +
 +  - Install/Configure Prometheus
 +  - Install/Configure Node Exporter
 +  - Install/Configure Nginx Reverse Proxy
 +  - Install/Configure Alert Manager
 +  - Install/Configure Blackbox Exporter
 +
 +
 =====Create rules===== =====Create rules=====
   sudo vim /etc/prometheus/uptime_rule.uml   sudo vim /etc/prometheus/uptime_rule.uml
Line 4: Line 16:
      
  
 +=====Blackbox Exporter=====
 +Blackbox Exporter by Prometheus allows probing over endpoints such as http, https, icmp, tcp and dns
 +
 +====Install the Blackbox Exporter====
 +https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
 +
 +  sudo useradd --no-create-home --shell /bin/false blackbox_exporter
 +  wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
 +  aunpack ./blackbox_exporter-0.19.0.linux-amd64.tar.gz
 +  sudo cp blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
 +  sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
 +  rm -rf blackbox_exporter-0.19.0.linux-amd64*
 +  # Config
 +  sudo mkdir /etc/blackbox_exporter
 +  sudo vim /etc/blackbox_exporter/blackbox.yml
 +<sxh bash; gutter: false>
 +modules:
 +  http_2xx:
 +    prober: http
 +    timeout: 5s
 +    http:
 +      valid_status_codes: []
 +      method: GET
 +      preferred_ip_protocol: "ip4" # used for "tcp", defaults to "ip4"
 +</sxh>
 +  
 +  sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
 +  sudo vim /etc/systemd/system/blackbox_exporter.service
 +<sxh bash; gutter: false>
 +[Unit]
 +Description=Blackbox Exporter
 +Wants=network-online.target
 +After=network-online.target
 +
 +[Service]
 +User=blackbox_exporter
 +Group=blackbox_exporter
 +Type=simple
 +ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml
 +
 +[Install]
 +WantedBy=multi-user.target
 +</sxh>
 +
 +  sudo systemctl daemon-reload
 +  sudo systemctl start blackbox_exporter
 +  sudo systemctl enable blackbox_exporter
 +====Configure Prometheus====
 +Next, we need to provide context to prometheus on what to monitor. We will inform prometheus to monitor a web endpoint on port 8080 using the blackbox exporter (we will create a python simplehttpserver to run on port 8080).
 +Edit the prometheus config /etc/prometheus/prometheus.yml and append the following:
 +  sudo vim /etc/prometheus/prometheus.yml
 +<sxh bash; gutter: false>
 +  - job_name: 'blackbox'
 +    metrics_path: /probe
 +    params:
 +      module: [http_2xx]
 +    static_configs:
 +      - targets:
 +        - http://localhost:8080
 +    relabel_configs:
 +      - source_labels: [__address__]
 +        target_label: __param_target
 +      - source_labels: [__param_target]
 +        target_label: instance
 +      - target_label: __address__
 +        replacement: localhost:9115
 +</sxh>
 +  
 +Open a new terminal, create a index.html:
 +  sudo pip3 install simple_http_server
 +  echo "ok" > index.html
 +  python -m SimpleHTTPServer 8080 or
 +  python3 -m http.server 8080
 +
 +  
 +Head back to the previous terminal session and restart prometheus:
 +  sudo systemctl restart prometheus
 +  
 +====Configure the Alarm definition:====
 +  sudo vim /etc/prometheus/blackbox_rule.yml
 +<sxh bash; gutter: false>
 +groups:
 +- name: alert.rules
 +  rules:
 +  - alert: EndpointDown
 +    expr: probe_success == 0
 +    for: 10s
 +    labels:
 +      severity: "critical"
 +    annotations:
 +      summary: "Endpoint  down"
 +</sxh>  
 +  
 +  sudo chown prometheus:prometheus /etc/prometheus/blackbox_rule.yml 
 +  promtool check rules /etc/prometheus/blackbox_rule.yml 
 +  sudo systemctl restart prometheus
 =====Setup Alert Manager===== =====Setup Alert Manager=====
   # Create the user for alertmanager   # Create the user for alertmanager
Line 36: Line 144:
   ## Access alertmanager on your endpoint on port 9093   ## Access alertmanager on your endpoint on port 9093
      
-   +To Debug: 
-  +  sudo journalctl -u alertmanager -f 
 + 
 +=====Telegram Bot for Alertmanager===== 
 +  https://github.com/nopp/alertmanager-webhook-telegram-python.git 
 +  sudo pip3 install -r ./requirements.txt 
 +  sudo pip3 install python-dateutil 
 + 
 + 
 +=====Issues===== 
 +====ICMP==== 
 +To get ping [icmp] to work, I had to do the following:   
 +  sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter  
 + 
prometheus.1633956980.txt.gz · Last modified: 2021/10/11 05:56 by vissie

Page Tools