ACA Blog

ACA Blog


April 2021
M T W T F S S
 1234
567891011
12131415161718
19202122232425
2627282930  

Categories


How to set up auto-discovery of Kubernetes endpoint services in Prometheus

Jeroen HartgersJeroen Hartgers

It’s kicking in an open door to state that you need monitoring when having applications running in Kubernetes (K8s). As K8s is becoming more mature and more adopted year after year (and here to stay), it’s only logical that lots of companies have built professional online monitoring and alerting solutions. Others have created excellent Helm charts that easily and swiftly install a K8s namespace with kube-state-metrics, Prometheus, Grafana with dashboards and Alert manager all-ready-to-go.

Scheme of infrastructure ready to go in blog post How to set up auto-discovery of Kubernetes endpoint services in Prometheus

The problem with that is that this is all rather ‘standard’. When you already have a fully customized Prometheus/Grafana setup in Rancher 1, such as we do, it seems a waste to throw this out the window. The journey from a Rancher 1 ‘cattle’ Prometheus/Grafana to Rancher 2 K8s went very smooth and was fairly easy.

However, with Prometheus, you historically would have to edit the prometheus.yaml file every time you want to scrape a new application, unless you had already added your own custom discovery tool as a scrape.

Fixing the incomplete data with auto-discovery

A problem that I faced with directly scraping a Longhorn and Spring Boot (or any other) Service in K8s, is that only one of the many backend pods behind that Service is scraped. So, you end up with incomplete data in Prometheus and hence incomplete data in your dashboards in Grafana. In Prometheus, you can see that only one of three existing Longhorn endpoints is scraped.

Image displaying that only one of three existing Longhorn endpoints is scraped in Prometheus.

In Grafana, you can see that there is only one node accounted for and the other two are reported as ‘Failed Nodes’. To make matters worse, only one of seven volumes is reported at ‘Total Number of Volumes’.

Image displaying Grafana dashboard showing only one node accounted for and the other two are reported as ‘Failed Nodes’.

This is where auto-discovery of Kubernetes endpoint services comes in as a true savior. Many web pages describe the various aspects of scraping, but I found none of them complete and others had critical errors.

In this blog post, I’ll provide you with a minimal and simple configuration to bring your Prometheus configuration with auto-discovery of Kubernetes endpoint services up to speed.

1. Include configMap additions for Prometheus

Add this to the end of the prometheus.yaml in your Prometheus configMap. The jobname is ‘kubernetes-service-endpoints’ as it seemed appropriate.

2. Configure the Services

As in the comment above of the prometheus.yaml, you can configure the following annotations. The annotation prometheus.io/scrape: “true” is mandatory, if you want to scrape a particular service. All the other annotations are optional and explained here:

Let’s look at an example for a Longhorn Service first. (Longhorn is a great replicated storage solution!)

Next, let’s look at an example for a Spring Boot Application Service. Note the non-standard scrape path /actuator/prometheus.

3. Configure Prometheus roles

ClusterRole

First, change the namespace as needed. Note: possibly this clusterRole needs to be a little tighter than it currently is.

ClusterRoleBinding

Again, change the namespace as needed.

ServiceAccount

Once more, change the namespace as needed. DO NOT change the name unless you change the ClusterRoleBinding subjects.name as well.

Apply

First, apply the ServiceAccount, ClusterRoleBinding, ClusterRole and Services to your K8s cluster. After updating the Prometheus configMap, redeploy Prometheus to make sure that the new configMap is activated/loaded.

Results in Prometheus

Go to the Prometheus GUI and navigate to Status -> Targets. You’ll see that now all the pod endpoints ‘magically’ pop up at the kubernetes-services-endpoints heading. Any future prometheus.io related annotation changes in k8s Services will immediately come into effect after applying them!

Image displaying Prometheus GUI with all the pod endpoints ‘magically’ popped up at the kubernetes-services-endpoints heading thanks to auto-discovery of Kubernetes endpoint services

Grafana Longhorn dashboard

I used a generic Grafana Longhorn dashboard, which you can find here for yourself. Thanks to the auto-discovery, the Grafana Longhorn dashboard now correctly shows three nodes and seven volumes, which is exactly correct!

Thanks to the auto-discovery of Kubernetes endpoint services, the Grafana Longhorn dashboard now correctly shows three nodes and seven volumes.

Conclusion

After running through all the steps in this blog post, you basically never have to look at your Prometheus configuration again. With auto-discovery of Kubernetes endpoint services, adding and removing Prometheus scrapes for your applications has now become almost as simple as unlocking your cell phone!

I hope this blog post has helped you out! If you have any questions, reach out to me using the comments below. Or, if you’d like professional advice and services, see how we can help you out with Kubernetes!


Hungry for more Kubernetes content? Check out these blog posts:

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x