ACA Blog

ACA Blog


September 2019
M T W T F S S
« Aug    
 1
2345678
9101112131415
16171819202122
23242526272829
30  

Categories


How to set up a secure logging platform with Elastic Stack and Search Guard

Jeroen HartgersJeroen Hartgers

You might roll your eyes at yet another blog about something as boring as log management with ELK. But that’s not really the focus of this blog post. Here, we dive deeper into the security aspect of an ELK stack and help you set up one yourself.

Log file chaos

Let’s say you are a system engineer/person involved in DevOps and have all these logs sitting on lots of permanent or temporary (cloud) servers. And then there are your colleagues: developers, auditors and regular users that need to read these logs or get reports. Giving all these people access to these logs will probably result in living hell, if it’s at all possible.

It gets even harder when the servers and their logs are part of a highly available environment, where there are lots of servers and lots of logs and it is even more difficult (read: impossible) to get logs from servers that no longer exist due to an autoscaling ‘scale in’ event. The solution is to use a log management system. A system like that gets the logs from all your servers, transforms and indexes them and finally stores them, ready for analysis and reporting.

Introduction to B+ELK a.k.a Elastic Stack

An Elastic Stack uses different ‘Beats’ to collect and ship the data to Logstash, where it is transformed and then pushed and stored in Elasticsearch. Then, everything is visualized in Kibana with nice graphs and filters so you can easily find what you need.

Secure logins and encrypted everything

An Elastic Stack is fairly easy to install with Docker. If you are using Rancher (for Docker container orchestration),  you can actually have a running Elastic Stack within just a few minutes. And it actually works as well, no funny business there.

However, little to no attention is paid to safety and security. Usually, when you look at an Elk Stack in a company, you can find it at an http address (so not encrypted), you do not have to log in (even guests of the network that stumble upon the server address can have a look at your log errors) and anyone can just manipulate the data inside Elasticsearch (ideal if you want to cover up failed login attempts)!

Both the open source Search Guard security plugin as well as X-Pack from Elastic.co (recently renamed to the less-catchy Elastic Stack Features) help make the Elastic Stack secure. An important difference is that X-Pack needs a paid license and Search Guard is open source for a lot of features and you can get a paid license when you need to integrate LDAP or AD and/or you need to control access to specific data. Simply looking at the cost, we choose to use the open source version of Search Guard, the ‘Community Edition’ at ACA IT-Solutions.

Setting up the entire stack

As a child, I almost exclusively played with Lego (Nintendo and internet didn’t really exist or were still pretty unheard of). The fun thing with LEGO is that you can combine all types of bricks and make something completely new. The only limit is your imagination!

When building this ELK platform, I looked at it in much the same way. I used the ELK Stack from Rancher 1.6 (see here) and replaced all the containers with docker Search-Guard container for Elasticsearch, Logstash and Kibana from  https://github.com/khezen.

For ElasticSearch management I picked Cerebro (Kopf does not work with Secure Elastic Stack) and last, Curator to clean up old indexes.

Traefik containers handle both the secure SSL traffic on TLS 1.2 and good Ciphers only (I dislike POODLE)  and the load balancing to the applications in the back. The Traefik containers are in turn connected to Amazon ALB’s for Kibana and Cerebro. Then I added a Beats stack with filebeat and metricbeat on all application servers to collect logs or other data. And also a homebrew log file cleaner to keep the logs themselves under control.

To sum it up:

This is what the entire stack looks like. Keep in mind that the Rancher Master server and Rancher Agent 3 are NEVER displayed (but they’re there for sure!).

Component list

TIP #1: There is a direct relationship between the amount of servers and logs that you feed to Elasticsearch and the disk storage that you use. The more servers, the more logs, the higher the amount of disk storage you need. However, if you clean indexes that you do not need correctly with Curator and you make sure your colleague devs and you try to keep the logs down (which is only good practice anyways), then the amount of disk you need can stay low. Bonus: it will also keep your ELK fast!

TIP #2: Keep all your shards for all indexes on all disks with a regular cron job insides. It keeps Elasticsearch healthier and possibly faster for longer at very little extra cost.

Rancher Stacks

On Amazon VPC – Rancher master with 3 Rancher agents

On VMWare internal servers – Rancher master with 3 Rancher agents

Certificates & passwords

A Search Guard script inside the khezen Elasticsearch container will create certificates on the first run. However, this did not seem to work well for me, as these certificates are created by isolated for each Elasticsearch container. So, this means that you have about nine servers creating their own (self-signed) certificates. The certificates are ‘only’ used for communication between all the Elastic Stack components, so self-signed certificates don’t hurt. Of course, some of the certificates need to be the same for all of the containers of the platform.

You can either fix the lack of certificate uniformity by:

  1. Putting the certificates inside the docker container, but this is rather unsafe.
  2. Making a script that only allows one of the containers to create these certificates (if they are not present from previous run) and share them through NFS with a container volume mount. This is what I did.
  3. Generating the certificates yourself and putting them in NFS.
  4. Putting your certificates in Rancher Secrets or Kubernetes Secrets. This will be a lot of work, as there are a LOT of certificates, but it is the safest and lowest maintenance option.

If you use Rancher and/or Kubernetes, then you’ll know that you can store passwords in Rancher Secrets and Kubernetes Secrets. You can then share them with the containers that need them.

Result: a secure logging platform

When you have done everything right, you should end up with a Kibana that is behind an Amazon ALB with certificates for https. You can then log in with the user account that you defined. Here’s an example of a Kibana dashboard:

Technical details

Below you can find all the technical details for:

Click the links to jump to the technical details for each component.

Amazon AWS logstash NLB

Load balancer – Listeners

Target group – Targets

Target group – Health Checks

Kibana Stack – docker-compose.yml

Logstash Stack – docker-compose.yml

Elasticsearch Stack – docker-compose.yml

BEATS Stack – docker-compose.yml