Monitoring of open and secure API endpoints using Prometheus Blackbox

In this blog, I am going to walk you through the steps on how to monitor open and secure API endpoints using the Prometheus Blackbox exporter.
Also, applying some standard Blackbox alerting rules to get notified users or systems when certain conditions are satisfied. Additionally, Grafana will be used for the visualization of the API endpoint's status.

The Blackbox exporter is a probing exporter used to monitor network endpoints such as HTTP, HTTPS, DNS, ICMP, or TCP endpoints.

prerequisites:

1. Prometheus should be installed in the system.
2. Grafana should be installed and configured with Prometheus.
3. The Alertmanager should be installed for notification.

Steps:

1. Install BlackBox Exporter on the system and configure it.

# Download prometheus Blackbox exporter
 wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz

# Configure prometheus Blackbox exporter
 tar xvzf blackbox_exporter-0.24.0.linux-amd64.tar.gz

# Rename the prometheus blackbox package
 sudo mv blackbox_exporter-*.linux-amd64 blackbox-package

#Go inside the blackbox-package folder
 cd blackbox-package

# Move the blackbox_exporter to /usr/local/bin/
 sudo mv blackbox_exporter /usr/local/bin/

# Create the Blackbox configurations workspace directory
 sudo mkdir /etc/blackbox

# Move the blackbox configuration in the configurations workspace directory
 sudo mv blackbox.yml /etc/blackbox/

# Create a user account for the Blackbox exporter
 sudo useradd --no-create-home --shell /bin/false blackbox

# Make sure that the blackbox binary can be run by your newly created user
 sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter

# Give the correct permissions to your configuration folders recursively
 sudo chown -R blackbox:blackbox /etc/blackbox/*

Now everything is set from an installation perspective.

2. Create a service file for the Prometheus Blackbox exporter.

# Move head to the /lib/systemd/system/ directory
 cd /lib/systemd/system/

# Create a blackbox.service file
 sudo touch blackbox.service

# Edit the blackbox.service file and paste the content mentioned below.
 sudo vim blackbox.service

[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox/blackbox.yml \
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target

3. Enable, Start, and verify the Blackbox service.

# Enable the blackbox service
 sudo systemctl enable blackbox.service

# Start the blackbox service
 sudo systemctl start blackbox.service


# Check status of the blackbox service
 sudo systemctl status blackbox.service

4. Modify the Blackbox configuration file and add modules to monitor open and secure API endpoints.

# Edit the blackbox.yml file, then clear and paste the below mentioned content
 sudo vim /etc/blackbox/blackbox.yml

modules:
  http_2xx:    # Open api endpoints
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET # post ,put ,delete

  http_2xx_auth:  # Secure api endpoints username and password
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: true
      basic_auth:
        username: "username"  # Replace username of the secure api endpoint
        password: "password"  # Replace password of the secure api endpoint

5. Bind the Blackbox exporter with Prometheus.

#Edit the prometheus.yml and add the blackbox specific jobs mentioned below
 sudo vim /etc/prometheus/prometheus.yml

  - job_name: 'blackbox-open-api-endpoints'
    metrics_path: /probe
    params:
      module: [http_2xx]  # pointingto open blackbox module
    static_configs:
      - targets:
        - https://api.openabc.com  # Target to probe with https.
        - https://app.opendef.com/actuator/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # The blackbox exporter's real hostname:port.

  - job_name: 'blackbox-secure-api-endpoints'
    metrics_path: /probe
    params:
      module: [http_2xx_auth]  #pointing to secure blackbox module
    static_configs:
      - targets:
        - https://api.securexyz.com/actuator/health    # Target to probe with https.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # The blackbox exporter's real hostname:port.

6. Restart, and verify the Blackbox and Prometheus service.

# Restart the blackbox service
 sudo systemctl restart blackbox.service

# Restart the promteheus service
 sudo systemctl restart prometheus.service

# Check status of the blackbox service
 sudo systemctl status blackbox.service

# Check status of the prometheus service
 sudo systemctl status prometheus.service

6. For the alerting of Blackbox endpoints alertmanger should be already installed in the Prometheus server.
Download the Blackbox standard alerting rules and specify that alerting rules file in the Prometheus configuration file.

# Install the standard blackbox alerting rules file
 wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/blackbox/blackbox-exporter.yml

# Remove and move the file
 sudo mv blackbox_exporter.yml /etc/prometheus/blackbox_alerts.yml

# Edit the promtheus.yml and add the alerts related configuration mendtioned below.
 cd /etc/prometheus/
 sudo vim promtheus.yml

Modify the prometheus.yml and add the below-mentioned content.

global:
 scrape_interval: 15s
rule_files:
 - /etc/prometheus/blackbox_alerts.yml
alerting:
 alertmanagers:
 - static_configs:
 - targets:
 - localhost:9093

6. Restart, and verify the Prometheus service.

# Restart the prometheus service
 sudo systemctl restart prometheus.service

# Check status of the prometheus service
 sudo systemctl status prometheus.service

7. In the Grafana console import the dashboard id 7587 to visualize the Blackbox endpoints.

Thank You