We will begin by understanding the principles of Elasticsearch, then go on to Kibana, KQL, and finally Grafana so you can begin monitoring your applications. Show
If you need full-text search Elasticsearch might be the right answer for you. [1] Instead of searching directly on the text, Elasticsearch searches through indexes and returns results quickly. Furthermore, it can perform statistical analyses and query scoring.[1] The inverted index [1] When saving data, Elasticsearch indexes the document containing a word. When we want to search for words, instead of searching through all of the data, we can quickly find the results by using the index list we created earlier.[1] Elasticsearch is an open-source distributed architecture project written entirely in Java and based on Apache Lucene. The Lucene project indexes and searches plain texts, which we refer to as mostly unstructured.[1] It can hold all the data that a relational database system can hold, as well as allow you to query this data much, much faster than a database system. It achieves this speed by employing special index structures that can store numerical values, geographical values, dates, and texts.[1] Elasticsearch decides how to keep the data and indexes for you, making it scalable. Stores data JSON document-oriented.[1] It can be used with Kibana, which can monitor Elasticsearch, and Logstash, which can host logs.[1] It is used with the rest API and can be used in any language. The default port is 9200. [2] http://localhost:9200/{indiceName}/_search?q={word to be searched} → Searches for the relevant word in all columns.[2] http://localhost:9200/{indiceName}/_search?q={field}:{word to be searched} → Searches the related word in the specific column. Highly scalable because it has clusters.[1] ClusterIt is a collection of multiple nodes that hold all of your data and perform all indexing and function searching.[1] ElasticSearch Cluster [2] NodeA node is a single real or virtual machine that stores data and performs computations. [3] Every node has a identifier. If no node identifier is specified, a random UUID is assigned at startup as a node identifier. [3] The property ‘cluster.name’ is present in every node configuration. The cluster will be formed automatically with all the nodes having the same `cluster.name` at startup.[3] Nodes must store data, keep the cluster healthy, and perform data operations. [3] Different Types of NodesThere are 3 types of nodes. [3] Data Node This node is capable of both storage and computation. Every node in the cluster is a data node by default. Part of the data is stored in the form of shards. [3] Master Node Master nodes monitor the availability and failure of data nodes and are also in charge of data creation and deletion. A cluster should have multiple master nodes.[3] Coordinate-Only Node These are the nodes that are not either data or master nodes. They are exposed to end-user requests in order to direct them between data nodes and master nodes, much like a smart load balancer.[3] Assume a request is sent to various data nodes. Each data node performs a local search and returns the results to the coordinating node. The Coordinating Node aggregates data and returns it to the user. An ElasticSearch Cluster [3] ShardA node divides an index to be created into shards as desired.[1] Advantages of sharding:It enables transaction distribution and parallelization across multiple nodes. Performance improves as a result. It lets you divide and scale the content volume horizontally.[1] ReplicaIf an index shard is disabled, a replica-shard structure allows for the creation of one or more copies.[1] Each node should keep a shard replica so that the search can be resumed if a node fails.[1] The Elasticsearch terms and their RDBMS term equivalents.[2] IndiceMultiple indices can exist in an ElasticSearch cluster.[1] TypeTables are referred to as types, and ElasticSearch indices can contain multiple types.[1] There are also some limitations to using types. Two fields with the same name in different types of documents should have the same data type.[3] DocumentRows are referred to as documents, and each type has multiple documents.[1] FieldColumns are referred to as fields. Every document has multiple fields.[1] Meta FieldsMeta fields are extra information about a document that is only used internally. The names of meta fields begin with an underscore. There are approximately ten meta-fields in total.[3] _index It saves the name of the index document. This is used internally to index and store the document.[3] _type It saves the document type. It is frequently included in search queries to improve performance.[3] _id This is the document’s unique identifier. It is used to directly access a specific document via the HTTP GET API.[3] _source This holds the original JSON document before any analyzers/transformations, etc. are applied. The source field is not indexed and thus cannot be queried, but it does appear in the final search result.[3] MappingIt is the process of specifying what type of data a data is while indexing.[1] If no mapping is provided, Elasticsearch attempts to identify the schema itself, which is referred to as Dynamic Mapping. [3] The Dynamic Mapping [1] Data TypesThere are three types of data types: simple, special, and complex.[3] Simple Data TypesText Full-text searching is possible in text types such as product descriptions.[3] Keywords These types are not saved in the same way that texts are. These fields are used in queries for filtering, aggregation, and sorting. Phone numbers, cities, ages, genders, and so on are some examples. [3] Numeric Elasticsearch accepts a variety of numeric types, including long, integer, short, byte, double, and float. [3] More data types are supported, including date (to store data in a variety of formats), boolean (true/false, on / off, 1 / 0), and IP. [3] Special Data TypesGeo point Geo points, which accept latitude and longitude pairs, are used to store geographical locations.[3] Geo shape Geo shapes are arbitrary geometric shapes such as rectangles and polygons that can be stored.[3] Completion Suggester The completion suggester is a feature that provides auto-completion over a specific field.[3] Complex Data typeObject Elasticsearch supports the storage of objects in JSON format.[3] Nested The Lucene index is not designed to store nested JSON objects. In ElasticSearch the nested, flattens the original JSON in order to store it in the Lucene index.[3] Installing ElasticsearchTo run Elasticsearch, you must first install Java. It should be at least version 8. Then set up Elasticsearch. You can get it from elastic.co and install it in your operating system by following the instructions. [4] KibanaA graphical user interface (GUI) is required to use Elasticsearch. Kibana is an open-source browser-based visualization tool that collaborates with Elasticsearch and Logstash to form the ELK stack. [4] The ELK Stack Kibana can be downloaded from “elastic.co”. Keep in mind that the versions of Elasticsearch and Kibana must match. [4] The default local address for Kibana is 5601. You can send your rest requests from the Dev Tools section. Kibana Query Language (KQL)In comparison to the Lucene language, this new language was designed to provide scripted field support and simplify the syntax.[6] Boolean OperatorsThe Boolean operators “AND” “and”, “OR” “or”, and “AND NOT” “and not” are all interchangeable in Kibana Query Language.[6] elasticsearch AND get The elasticsearch AND get query result. [7] Using AND with fields: machine.os.keyword : "win xp" AND response.keyword : "404" All results that both machine.os.keyword : "win xp" AND response.keyword : "404" 9 and machine.os.keyword : "win xp" OR response.keyword : "404" 0 appear together [7] Using OR with field values: machine.os.keyword : "win xp" OR response.keyword : "404" all results where the OS is Windows XP, or the response was machine.os.keyword : "win xp" OR response.keyword : "404" 1 Instead of “NOT”, you need to write “AND NOT” or “and not”.[6] Moira AND NOT Johnny By default, “and” has priority over “or.” To override this default, use parentheses.[6] David and (**Patrick **or Stevie*)* Putting “not” before a search query invert its meaning.[6] not location:”Blouse Barn” Parentheses can be used to invert entire groups.[6] not (name:Patrick or location:”Rose Apothecary”) Searching by Fieldname: Twyla The value “Bob’s Garage” is enclosed in quotation marks to ensure that the search includes the words Bob’s and Garage in that order. Without the quotation marks, the results would include Garage Bob’s as well.[6] location: “Bob’s Garage” Individual fields can be searched for in Kibana’s bottom left menu pane under “Available Fields”. [6] Available fields [7] To search multiple fields[6]: location: (“Bob’s Garage” or “Café Tropical”) or machine.os.keyword : "win xp" AND response.keyword : "404" 0 or machine.os.keyword : "win xp" AND response.keyword : "404" 1All matched instances in the specified field[7] RangesA range is frequently required for numeric and date types. KQL supports the four range operators listed below:
machine.os.keyword : "win xp" AND response.keyword : "404" 2All dates before and including the listed date.[7] For searches, colons are removed before greater than, less than, and so on.[6] machine.os.keyword : "win xp" AND response.keyword : "404" 3 Wildcard(*)Used for unknown characters.[6] machine.os.keyword : "win xp" AND response.keyword : "404" 4 Existing queries are executed by appending an asterisk to the field name. Following the query, the value of the room number field should exist.[6] machine.os.keyword : "win xp" AND response.keyword : "404" 5 Wildcards can be used to search both the text and keyword versions of a field at the same time. We’re looking for both the plot and the plot keyword. fields being Galápagos.[6] machine.os.keyword : "win xp" AND response.keyword : "404" 6 You can use a free text search to find a word in any field by simply writing it. [7] machine.os.keyword : "win xp" AND response.keyword : "404" 7Free text search results [7] or machine.os.keyword : "win xp" AND response.keyword : "404" 8 machine.os.keyword : "win xp" OR response.keyword : "404" 2 locates machine.os.keyword : "win xp" OR response.keyword : "404" 3 and machine.os.keyword : "win xp" OR response.keyword : "404" 4 as separate words. [7] To match an exact string, use quotation marks[7]: machine.os.keyword : "win xp" OR response.keyword : "404" 5 queries the whole string.[7] Kibana FilterThe Kibana filter assists in excluding or including fields from search queries.[7]
To create the filter, a dialog box appears.[7] 2. To get auto-suggestions, select a Field from the dropdown menu or begin searching.[7] 3. From the dropdown menu, select an Operator.[7] 4. Depending on the operator selected, an additional Value field appears. All other operators require the value field except the exists and does not exist options. Choose the filtering value if the operator needs it.[7] 5. Create a custom label for the filter as an optional step. Tick the Create custom label box and give it a name. To finish, click Save.[7] Edit filter screen.[7] The filter appears below the search box and applies to current data and all further searches automatically. You can apply multiple filters to further narrow the dataset search. [7] The search filter box. [7] Kibana Index PatternA defined index pattern instructs Kibana on which Elasticsearch data to retrieve and use. [7] To add an index pattern:
3. Select the “create index pattern” option.[7] The create index pattern button.[7] 4. To define the index pattern, look up the index by its exact name. For a close match or to match multiple indexes with the same name, use an asterisk (*). [7] Index pattern matches.[7] If you are unclear about the index name, a list of available index patterns is provided at the bottom. For the tutorial, we’re going to use the Kibana sample web traffic data. To proceed, click Next step.[7] 5. If the data contains an index with a timestamp, specify the default time field for time filtering. From the dropdown menu, select the appropriate option.[7] Selecting time field.[7] If you don’t have time data or want to merge time fields, choose I don’t want to utilize the time filter. To finish, click the Create index pattern button.[7] 6. Enter Discover in the search bar (CTRL+/) to explore the data.[7] 7. Select the index pattern from the left pane’s drop-down menu.[7] The data from the created index pattern is displayed on the discover page.[7] GrafanaGrafana is Grafana Labs’ open-source interactive data visualization platform. It enables users to view their data through charts and graphs that are combined into dashboards for easier interpretation and comprehension. [8] Grafana lets you query, alert, visualize, and explore your metrics regardless of where they are stored. It also allows us to track user behavior, application behavior, the frequency of errors in the production or pre-production environment, and so on. [8] The best thing about Grafana, however, is that it can connect to any data source. [8] Grafana starts working at (http://localhost:3000/) by default after being downloaded and installed from its website (https://grafana.com/). When you first launch the application, you can log in by entering “admin” in the username and password fields on the login screen. After logging in, you will be forced to change your password. [5] Grafana login screen [7] Grafana can be used to;
It also has visualization options such as geo maps, heat maps, histograms, and so on. A dashboard is made up of separate grid panels. Each panel has a distinct purpose. The main graphical building block is the panel. However, each panel has a query editor tailored to the data source selected in the panel. Above all, the query editor enables you to extract the ideal visualization to show on the panel. Panels can be styled and formatted in a variety of ways. Panels on the dashboard can be dragged and dropped and rearranged. They are also resizable. [8] Features of GrafanaPanels like heatmaps, histograms, and graphs are quick and flexible visualizations that let you visualize your data however you want. [8] Meanwhile, Grafana plugins can connect your tools and teams. Furthermore, data source plugins connect to existing data sources via APIs and render the data in real-time, eliminating the need to migrate or ingest your data. [8] Similarly, with alerting, you can create, manage, and quiet all of your alerts from a single UI. This allows you to concentrate and aggregate all of your alerts. [8] Furthermore, transformations allow you to rename, summarize, combine, and perform calculations across multiple queries and data sources. [8] Additionally, annotate graphs with rich events from various data sources. Hover over an event to see the full event metadata and tags. [8] Above all, the panel editor makes it simple to configure, customize, and explore all of your panels with a unified user interface. You can use it to configure data options across all of your visualizations. [8] How to use Grafana?The first step in creating your Grafana dashboard is to configure your data source. Dashboards are made up of information panels organized into rows. Panels are graphical representations of queries. Each panel can display the same or different data using the most user-friendly visualization. [9] We create a “Dashboard” to visually display our data by clicking the “Create” button on the left side menu. On the next screen, we click the “Add New Panel” button to return to the main area. When we click, the screen below appears [5]: New Dashboard Screen [5] All we need to do for my currently empty panel is type a query in the A field. We can create and modify a simple SQL query to display data. Writing a query for a panel [9] Expand the Visualization menu on the right-hand side of the panel tab and select the “stat” icon. This will immediately provide a preview of your panel with the total number of records in your table. [5] Finally, select the settings icon to name the panel something significant. You can choose how frequently the database will be requested to acquire up-to-date information at the top right of the screen. Setting it to 5s will cause the box to update every 5 seconds. [5] Visuals of the sql queries [5] How to export my dashboard on Grafana?To export your dashboard, click the cog icon in the top right corner. Dashboard Settings will be displayed in the tooltip. Select JSON Model from the left navigation menu to view your dashboard in JSON format. It is as simple as saving this file and sending it to someone else to load into their instance to share your dashboard. [9] Exporting Dashboard [9] Thank you for reading. Please do not forget to clap if you think it was helpful. Resources[1] Kadircan Doğan, (3, Feb, 2019) ,Elasticsearch nedir? [https://medium.com/@kdrcandogan/elasticsearch-nedir-45d237c29b26] [2] Fatma Delen,(17, Jan, 2021), Elasticsearch nedir? Nasıl kullanılır? [https://fatmadelenn.medium.com/elasticseacrh-nedir-nas%C4%B1l-kullan%C4%B1l%C4%B1r-d99c28b3e779] [3] Velotio Technologies, (), Elasticsearch 101: Fundamentals and Core Components: [https://medium.com/velotio-perspectives/elasticsearch-101-fundamentals-core-components-a1fdc6090a5e] What is the difference between Grafana and Elasticsearch?Elasticsearch is an enterprise search tool from Elastic in Mountain View, California. Grafana is a data visualization tool developed by Grafana Labs in New York. It is available open source, managed (Grafana Cloud), or via an enterprise edition with enhanced features. How do I monitor Elasticsearch cluster with Grafana?Go to: http://localhost:3000/dashboard/import and enter 14191 as the Dashboard ID, and click Load. Now go to http://localhost:3000/dashboards and select: Elasticsearch Exporter Quickstart and Dashboard. Done! Now users can monitor their Elasticsearch instance using Prometheus and Grafana. What is the limit of Elasticsearch in Grafana?Logs queries analyze Elasticsearch log data. You can configure the following options: Logs Options/Limit - Limits the number of logs to analyze. The default is 500 . Is elastic APM better than Grafana?Elastic APM rates 4.0/5 stars with 13 reviews. By contrast, Grafana Labs rates 4.5/5 stars with 121 reviews. Each product's score is calculated with real-time data from verified user reviews, to help you make the best choice between these two options, and decide which one is best for your business needs. |
