Elasticsearch has increasingly been gaining momentum within the security community. This growth was driven at first by the ease and scale at which log data can be ingested and searched and has been further fuelled by Elastic’s own security offerings which include out of the box capabilities for anomaly detection, endpoint detection and responses, as well as a SIEM offering. Given that at Satisnet we already have expertise within the fields of SIEM deployment and staffing as well as with a multitude of security solutions and their associated logs, Elasticsearch provides a fantastic “glue” with which to link many of our other offerings. To support this, we have been looking to continue to build our Elasticsearch expertise internally.
We have been using Elasticsearch for quite some time now, both internally and to support clients, so we’re far from newcomers to the Elastic scene. We decided that to show our expertise in the product it would be valuable to get certified through Elastic’s Certified Engineer program. Elastic publish the exam objectives for the certification on their website, and they recommend taking their two Elastic engineer classes in order to prepare for the certification exam. The exam consists of a series of real-world exercises that you complete with the aid of the Elastic documentation.
We looked over the exam objectives and while we felt comfortable with most of them we decided that it would be worth completing the Elastic training just to identify any key gaps in our knowledge and make sure we were following any recommendations when it comes to best practices. The training that Elastic offers comes as two classes, which can either be taken in-person or virtually. The virtual classes are four days each and the in-person training amounts to two days each for both Engineer I and II.
We were lucky enough to find classroom training in Edinburgh where we’re based so we opted for doing Engineer I and II over a four-day period. It’s worth noting that this training is specifically around Elasticsearch, and thus doesn’t touch much on Logstash, Beats, or Kibana (although a certain amount of Kibana comes into it as Kibana is used as the interface between the user and Elasticsearch).
The first two days of the course were not too intense – especially given that we already went in with a fair amount of Elastic experience. We started off by going through the very basics of what an Elastic cluster is and how you can query the data in it through basic queries and aggregations. We then moved into a section on text analysis followed by a final overview of nodes and shards and general cluster makeup and a basic guide to monitoring and troubleshooting Elasticsearch. This material was broken down into sections with a few lessons per topic. Each lesson came with an accompanying lab where we were given access to a cloud lab environment where we spun up and played around with out own Elasticsearch cluster. The labs were good at emphasising the important parts of the lesson and given that the certification exam is based on practical exercises they were worth focussing on. By and large, there was enough time to complete the labs for the first couple of days. Although some required a second look in the evening after class.
Elastic Engineer II was where out Elasticsearch experience was really tested. The second two day course went into far more detail than Engineer I and it was a sharp rise in terms of the quantity of new information that we had to take in. We delved into how to model your data to best be used in Elasticsearch and looked at how to represent some more complex data structures in Elasticsearch. We then went on to look at ingest pipelines and how to process data via Elasticsearch. We went on to go over important considerations when deploying a cluster, including information on how to manage data across nodes and how to manage the lifecycle of your data. We finished off by looking at some more general tips and tricks. It’s worth noting that during this second course the allocated time for some of the labs started to get tight. Whereas in the first two day course I hadn’t had much of an issue getting through the labs in the time set aside for them in the class, during Engineer II I definitely found myself having to mark things to go back and look at more carefully as I wasn’t able to go through all of the required lab exercises in time. These more complicated labs did mean that I had more questions for the instructors though who were extremely knowledgeable.
Overall the three of us who took the Engineer I and II training came out of it feeling like it had been immensely valuable for us. Engineer I definitely was mostly a recap for us, but it still felt like it had been worthwhile – there were quite a few new things that we learned and it set us straight on some best practices. Engineer II was where we definitely found the most value – there were a lot of concepts that we hadn’t really looked at in any detail before and it certainly made us aware of a lot of different (and more elegant) ways of doing things than what we had been doing beforehand. This did however lead to us feeling pretty “frazzled” after the end of the last day due to the amount of information we had to digest.
Due to this, I think were I to recommend the course to anyone I would make sure to recommend them to have some Elastic experience going in to the four day course, or to split up the two courses a few months apart. Even though the two classes start off assuming very little experience I think it would be really hard to get much value from the class were you to try and use the four day course to take you from zero to hero. The other option to look into if time is a concern and you don’t like the idea of being rushed (particularly on the labs) would be doing the course virtually. You get access to the same environment and instructors but the course is spread over 4 days, meaning there’s a little more time to digest information between lessons.
In summary we took away a lot of interesting lessons from the training and we are really looking forward to implementing everything that we learned and using our foundation from the course to become true Elastic experts. Thanks very much to Lizzie and Daniel from Elastic for providing such a fun week for us!
By Calum Finlayson, Satisnet