=====The order of things=====
  * https://blog.ruanbekker.com/blog/2019/05/07/setup-prometheus-and-node-exporter-on-ubuntu-for-epic-monitoring/
  * https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/
  * https://www.scaleway.com/en/docs/tutorials/prometheus-monitoring-grafana-dashboard/

  - Install/Configure Prometheus
  - Install/Configure Node Exporter
  - Install/Configure Nginx Reverse Proxy
  - Install/Configure Alert Manager
  - Install/Configure Blackbox Exporter


=====Create rules=====
  sudo vim /etc/prometheus/uptime_rule.uml
  ..add your rule code here...
  

=====Blackbox Exporter=====
Blackbox Exporter by Prometheus allows probing over endpoints such as http, https, icmp, tcp and dns

====Install the Blackbox Exporter====
https://blog.ruanbekker.com/blog/2019/05/17/install-blackbox-exporter-to-monitor-websites-with-prometheus/

  sudo useradd --no-create-home --shell /bin/false blackbox_exporter
  wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
  aunpack ./blackbox_exporter-0.19.0.linux-amd64.tar.gz
  sudo cp blackbox_exporter-0.19.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
  sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
  rm -rf blackbox_exporter-0.19.0.linux-amd64*
  # Config
  sudo mkdir /etc/blackbox_exporter
  sudo vim /etc/blackbox_exporter/blackbox.yml
<sxh bash; gutter: false>
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4" # used for "tcp", defaults to "ip4"
</sxh>
  
  sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
  sudo vim /etc/systemd/system/blackbox_exporter.service
<sxh bash; gutter: false>
[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target
</sxh>

  sudo systemctl daemon-reload
  sudo systemctl start blackbox_exporter
  sudo systemctl enable blackbox_exporter
====Configure Prometheus====
Next, we need to provide context to prometheus on what to monitor. We will inform prometheus to monitor a web endpoint on port 8080 using the blackbox exporter (we will create a python simplehttpserver to run on port 8080).
Edit the prometheus config /etc/prometheus/prometheus.yml and append the following:
  sudo vim /etc/prometheus/prometheus.yml
<sxh bash; gutter: false>
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://localhost:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
</sxh>
  
Open a new terminal, create a index.html:
  sudo pip3 install simple_http_server
  echo "ok" > index.html
  python -m SimpleHTTPServer 8080 or
  python3 -m http.server 8080

  
Head back to the previous terminal session and restart prometheus:
  sudo systemctl restart prometheus
  
====Configure the Alarm definition:====
  sudo vim /etc/prometheus/blackbox_rule.yml
<sxh bash; gutter: false>
groups:
- name: alert.rules
  rules:
  - alert: EndpointDown
    expr: probe_success == 0
    for: 10s
    labels:
      severity: "critical"
    annotations:
      summary: "Endpoint  down"
</sxh>  
  
  sudo chown prometheus:prometheus /etc/prometheus/blackbox_rule.yml 
  promtool check rules /etc/prometheus/blackbox_rule.yml 
  sudo systemctl restart prometheus
=====Setup Alert Manager=====
  # Create the user for alertmanager
  useradd --no-create-home --shell /bin/false alertmanager
  # Download alertmanager and extract
  wget https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz
  aunpack ./alertmanager-0.23.0.linux-amd64.tar.gz 
  # Move alertmanager and amtool birnaries in place
  sudo cp ./alertmanager-0.23.0.linux-amd64/alertmanager /usr/local/bin/
  sudo cp ./alertmanager-0.23.0.linux-amd64/amtool /usr/local/bin/
  # Ensure that the correct permissions are in place
  sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
  sudo chown alertmanager:alertmanager /usr/local/bin/amtool
  # Cleanup
  rm -rf alertmanager-0.23.0.linux-amd64*
  # Configure Alertmanager
  ## Create the alertmanager directory and configure the global alertmanager configuration
  sudo mkdir /etc/alertmanager
  vim /etc/alertmanager/alertmanager.yml
  # Ensure the permissions are in place
  chown alertmanager:alertmanager -R /etc/alertmanager
  # Create the alertmanager systemd unit file
  sudo vim /etc/systemd/system/alertmanager.service
  # Now we need to inform prometheus that we will send alerts to alertmanager to it’s exposed port
  sudo vim /etc/prometheus/prometheus.yml
  sudo vim /etc/systemd/system/prometheus.service
  sudo systemctl daemon-reload
  sudo systemctl restart alertmanager.service 
  sudo systemctl restart prometheus.service 
  sudo systemctl status prometheus.service
  sudo systemctl status alertmanager.service 
  ## Access alertmanager on your endpoint on port 9093
  
To Debug:
  sudo journalctl -u alertmanager -f

=====Telegram Bot for Alertmanager=====
  https://github.com/nopp/alertmanager-webhook-telegram-python.git
  sudo pip3 install -r ./requirements.txt
  sudo pip3 install python-dateutil


=====Issues=====
====ICMP====
To get ping [icmp] to work, I had to do the following:  
  sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter