Elasticsearch is the central component of the Elastic Stack, a set of open-source tools for data ingestion, enrichment, storage, analysis, and visualization. It is commonly referred to as the “ELK” stack after its components Elasticsearch, Logstash, and Kibana and now also includes Beats.
Although a search engine at its core, users started using Elasticsearch for log data and wanted a way to easily ingest and visualize that data.
At its core, you can think of Elasticsearch as a server that can process JSON requests and give you back JSON data.
How do Beats work?
Beats is a collection of lightweight, single-purpose data shipping agents used to send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch. Beats are great for gathering data as they can sit on your servers, with your containers, or deploy as functions then centralize data in Elasticsearch. For example, Filebeat can sit on your server, monitor log files as they come in, parses them, and import into Elasticsearch in near-real-time.
Beats is a collection of data shipper light weight agents which send specific metric data about the server in which they are installed on. Few of the popular beat agents are:
-
Filebeat: Used for collecting log files and sending them to logstash or elastic search.
-
Metricbeat: Used for collecting system level operational data like CPU, memory usage etc, and send it to logstash or elastic search.
-
Heartbeat: Monitors service uptime
-
Auditbeat: Collects audit data from Linux servers.
-
Winlogbeat: Collects Windows Events logs.
-
Packetbeat: Collects network data
How does Elasticsearch work?
Raw data flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications.
Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch. Once indexed in Elasticsearch, users can run complex queries against their data and use aggregations to retrieve complex summaries of their data.
From Kibana, users can create powerful visualizations of their data, share dashboards, and manage the Elastic Stack.
What is an Elasticsearch Index?
An Elasticsearch index is a collection of documents that are related to each other. Elasticsearch stores data as JSON documents. Each document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data).
Elasticsearch uses a data structure called an inverted index, which is designed to allow very fast full-text searches. An inverted index lists every unique word that appears in any document and identifies all of the documents each word occurs in.
During the indexing process, Elasticsearch stores documents and builds an inverted index to make the document data searchable in near real-time. Indexing is initiated with the index API, through which you can add or update a JSON document in a specific index.
What is Logstash used for?
Logstash, one of the core products of the Elastic Stack, is used to aggregate and process data and send it to Elasticsearch. Logstash is an open source, server-side data processing pipeline that enables you to ingest data from multiple sources simultaneously and enrich and transform it before it is indexed into Elasticsearch.
What is Kibana used for?
Kibana is a data visualization and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, and maps. Kibana also includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.
-
Guys, what do you think of this post on -- how does Elasticsearch work?
Kindly leave me your thoughts in the comment section.
This Article Was Written & published by Meena R, Senior Manager - IT, at Luminis Consulting Services Pvt. Ltd, India.
Over the past 16 years, Meena has built a following of IT professionals, particularly in Cybersecurity, Cisco Technologies, and Networking...
She is so obsessed with Cybersecurity domain that she is going out of her way and sharing hugely valuable posts and writings about Cybersecurity on website, and social media platforms.
30,000+ professionals are following her on Facebook and mesmerized by the quality of content of her posts on Facebook.
If you haven't yet been touched by her enthusiastic work of sharing quality info about Cybersecurity, then you can follow her on Facebook:
Click Here to follow her: Cybersecurity PRISM