Kubernetes (AWS EKS) Alerting Using Prometheus

Set up Kubernetes (AWS EKS), standard alerting using Prometheus and then send the alerts to Slack channel using Prometheus-Alertmanager.

In this blog, I am going to walk you through the steps on how to set up Kubernetes (AWS EKS ) standard alerting using Prometheus and then send the alerts to the slack channel using Prometheus-Alertmanager.
This blog covers all the standard alerting rules for AWS EKS cluster monitoring.

Steps:

Install the Metrics server in AWS EKS.

#Deploy the Metrics Server with the following command:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

#Verify that the metrics-server deployment is running the desired number of Pods with the following command.
kubectl get deployment metrics-server -n kube-system

2. Install Prometheus in AWS EKS using Helm.

# Add helm repositories for Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Install Prometheus
helm upgrade -i prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.persistence.storageClass="gp2",server.persistentVolume.storageClass="gp2"

#Verify the deployment, statefulsets, services, and configmaps get created in prometheus namespace
kubectl get deploy,service,configmap,statefulset -n prometheus

3. verify the services in the namespace prometheus and port-forward the prometheus-server service to access the Prometheus console temporarily.

# port-forward the promtheus-server service to access the prometheus console
kubectl port-forward svc/prometheus-server -n prometheus 9090:80

Open the browser and type localhost:9090 in the search bar. Click on the Alerts button and see the current alerts configured. By default, no alerts are set.

4. In order to set up Kubernetes (AWS EKS) Cluster-related alerts, edit the configmap/prometheus-server in prometheus namespace.

Then add the custom alerts that I have mentioned in this GitHub repository in the data alerting_rules.yml section, with proper indentation(refer to sample-prometheus-server-configmap.yaml).

Save the changes and restart the deployment/promtheus-server .

# Restart the prometheus-server deployment
kubectl rollout restart deploy prometheus-server -n prometheus

Go to the console and check the configured alerts in the alerts section.

5. Configure the prometheus-alermanager for slack notifications.
Create a slack channel and in apps sections on the slack app search the Incoming WebHooks. add the Incoming WebHooks.

then configure and integrate with the Slack notification channel that you have created. Finally, get the Webhook URL and keep it handy.

6. Edit the configmap/prometheus-alertmanager in prometheus namespace and in the data alertmanager.yml section add the alertmanager.yml configuration that I have mentioned in the Github repository .

For proper indentation check the sample-prometheus-alertmanager-configmap.yaml file. Make sure you have specified your own Webhook URL in the file.

Save the configmap/prometheus-alertmanager and restart the statefulset.apps/prometheus-alertmanager.

# Restart the prometheus-alertmanager statefulset
kubectl rollout restart statefulset prometheus-alertmanager -n prometheus

Now finally you will see the alert notifications on the Slack channel.

Thank you.

Further Reading:

Serverless observability, monitoring, and debugging explained

Kubernetes Monitoring with OpenTelemetry

Top Tools for AWS Lambda Monitoring

OpenTelemetry: A full guide

9 Best Distributed Tracing Tools for Developers