Elasticsearch Stack is something that might elude even the most seasoned programmers. This is mainly because we programmers tend to look at things from above. But in the case of Elasticsearch Stack, we need to start from the bottom and learn what it really is, and how it came to be what it is today.
As an overview, so that programmers don't lose interest, Elasticsearch Stack or "ELK" in short is an acronym for no less than three open source projects:
Logstash is a data processing pipeline on the server side, it has the ability to process data simultaneously from multiple sources. Logstash transforms this data and sends it to Elasticsearch. Logstash is the first layer. Let's move on to the second layer.
Elasticsearch is a search and analytics engine for Big Data, more or less what Google is for Web Pages. It takes the "Stash" forwarded by Logstash and performs analysis on the data to get meaningful results.
Kibana is the third layer of this stack. Kibana let's user represent the finding of Elasticsearch with the help of charts, graphs, and a dozen more data visualization techniques.
Now that we have the overall understanding out of the way. Let's move on to each layer of the stack and dissect them one by one.
It is important that you don't take ELK as just another way to deal with Big Data. It is much, much more than that.
In a short amount of time, it has grown into something very unique, and it all started with the introduction of Elasticsearch.
The First Phase
Elasticsearch, as mentioned before, is an open source project. It is not just that, but also distributed, RESTful, JSON-based search engine. It is extremely easy to use and effectively scales accordingly. It also allows for unique flexibility that was nonexistent before.
Soon after it was launched it gained immense traction. Within a short amount of time, it developed a huge following.
Following the success, this open-source project turned into a full-fledged company.
The Second Phase
Enter Logstash and Kibana. Since the project was open-source the general population was able to integrate other tools with it. In the end, the most successful and beneficial integration turned out to be of Logstash and Kibana.
This integration resulted from the need of entering logs to the engine and representing the finding with data visualization like charts, graphs and so on.
The Third And Final Phase
In this phase, the final product which we know as ELK was formalized. It became impossible to think of any of these three open-source projects as a single entity. Now, they were all parts of a single unit ELK.
With this combination, no matter how thick was the jungle of text and no matter how difficult was traversing it. ELK did it with extreme accuracy and represented the data in a visually appealing manner.
You can find the top N results. You can freely chop the metrics. No matter what you wanted to do the community of developers was able to find a way to do it. Each and everyday developers pushed the limits of this stack.
And this it became what it is today.
The Difficult PartNow, let's talk about the difficulty of understanding that I talked about before. The difficulty of understanding arises because three different entities are acting as one unit. It is impossible to work with ELK without first mastering each one separately.
And this is all that is in the way of learning ELK.
You can learn how to set up the ELK in pretty easily.
Where Is It HeadedThe future ELK, according to the official website, is full of acronym sophistication. Just like the addition of Logstash and Kibana, there are more and more potential integrations that are formally under consideration. One of them which has been formally added is "Beats".
Beats allow a user to tail a file and its interactions in the jungle of documents and unimaginable data.
With the addition of Beats, no one knows what to call Elasticsearch Stack anymore. Should we call it ELKB or BELK or KELB?
And with this, I leave you an eternal mystery... until someone solves it in a way everyone accepts.
