Got it

[FI Components] Working principles of Elasticsearch

Latest reply: Apr 12, 2022 06:41:13 962 6 3 0 0

Hello all,

Does anyone know the working principles of Elasticsearch? Don't worry, please find out below.

Internal architecture

The Elasticsearch provides various access interfaces through RESTful APIs or other languages (such as Java), uses the cluster discovery mechanism, supports script languages, and various plug-ins. The underlying layer is based on Lucene, with absolute independence of Lucene and stores indexes through local files, shared files, and HDFS, as shown in Figure 2.

Figure 2 Internal architecture
Elasticsearch1

Inverted indexing

In the traditional search mode, as shown in Figure 3, documents are searched based on their IDs. During the search, keywords of each document are scanned to find all information containing the keywords. Forward indexing is easy to maintain but is time consuming.

Figure 3 Forward indexing
Elasticsearch2

For Elasticsearch (Lucene), as shown in Figure 4, there is a dictionary composed of keywords and their statistic information, such as IDs, positions, and document frequencies. In this search mode, Elasticsearch searches for the keywords to locate the document ID and position and then finds the document, which is similar to the method of looking up a word in a dictionary. Alternatively, Elasticsearch searches the catalog to find the content on a specific page. Inverted indexing is efficient in search, but is time consuming for constructing indexes and costly for maintenance.

Figure 4 Inverted indexing
Elasticsearch3

Distributed indexing flow

Figure 5 shows the process of Elasticsearch distributed indexing flow.

Figure 5 Distributed indexing flow
Elasticsearch4

The process is as follows:

Phase 1: The client sends an index request to any node, for example, Node 1.

Phase 2: Node 1 determines the shard to store the file based on the request. Assume the shard is shard 0. Node 1 then forwards the request to Node 3 where primary shard P0 of shard 0 exists.

Phase 3: Node 3 executes the request on primary shard P0 of shard 0. If the request is successfully executed, Node 3 sends the request to all the replica shard R0 in Node 1 and Node 2 concurrently. If all the replica shards successfully execute the request, a verification message is returned to Node 3. After receiving the verification messages from all the replica shards, Node 3 returns a success message to the user.

Distributed searching flow

The Elasticsearch distributed searching flow consists of two phases: Query and acquisition.

Figure 6 shows the query phase.

Figure 6 Query phase of the distributed searching flow
Elasticsearch5

The process is as follows:

Phase 1: The client sends a retrieval request to any node, for example, Node 3.

Phase 2: Node 3 sends the retrieval request to each shard in the index adopting the polling policy. One of the primary shards and all of its replica shards is randomly selected to balance the read request load. Each shard performs retrieval locally and adds the sorting result to the local node.

Phase 3: Each shard returns the local result to Node 3. Node 3 combines these values and performs global sorting.

In the query phase, the data to be retrieved is located. In the acquisition phase, these data will be collected and returned to the client. Figure 7 shows the acquisition phase.

Figure 7 Acquisition phase of the distributed searching flow
Elasticsearch7

The process is as follows:

Phase 1: After all data to be retrieved is located, Node 3 sends a request to related shards.

Phase 2: Each shard that receives the request from Node 3 reads the related files and return them to Node 3.

Phase 3: After obtaining all the files returned by the shards, Node 3 combines them into a summary result and returns it to the client.

Distributed bulk indexing flow

Figure 8 Distributed bulk indexing flow
Elasticsearch8

The process is as follows:

Phase 1: The client sends a bulk request to Node 1.

Phase 2: Node 1 constructs a bulk request for each shard and forwards the requests to the primary shard according to the request.

Phase 3: The main segment is executed one by one in sequence. After an operation is complete, the main segment forwards the new document (or deletes the part) to the corresponding replication node, and then performs the next operation. After the operation is complete, the node sends a report to the request node. The request node sorts the response and sends the response to the client.

Distributed bulk searching flow

Figure 9 Distributed bulk searching flow
Elasticsearch9

The process is as follows:

Phase 1: The client sends an mget request to Node 1.

Phase 2: Node 1 constructs a multi-data retrieval request for each shard and forwards the requests to the primary shard or its replica shard based on the requests. When all replies are received, Node 1 constructs a response and returns it to the client.

Routing algorithm

Elasticsearch provides two routing algorithms:

§  Default route: shard=hash (routing) %number_of_primary_shards. In this routing policy, the number of received shards is limited. During capacity expansion, the number of shards needs to be multiplied (ES6.x). In addition, when creating an index, you need to specify the capacity to be expanded in the future. ES5.x does not support capacity expansion. ES7.x can be expanded freely.

§  Custom route: In this routing mode, the routing can be specified to determine the shard to which the file is written, or only the specified routing can be searched.

Balancing algorithm

Elasticsearch provides the automatic balance function for capacity expansion, capacity reduction, and data import scenarios. The algorithm is as follows:

weight_index(node, index) = indexBalance * (node.numShards(index) - avgShardsPerNode(index))

Weight_node(node, index) = shardBalance * (node.numShards() - avgShardsPerNode)

weight(node, index) = weight_index(node, index) + weight_node(node, index)

Single-node multi-instance deployment

Multiple Elasticsearch instances can be deployed on the same node, and differentiated from each other based on the IP address and port number. This method increases the usage of the single-node CPU, memory, and disk, and improves the Elasticsearch indexing and searching capability.

Elasticsearch10

Cross-node replica allocation policy

When multiple instances are deployed on a single node and multiple replicas exist, replicas can only be allocated across instances. A single-point failure may occur. Solve this problem by configuring the parameter cluster.routing.allocation.same_shard.host to true.

11

 

More content about Bigdata, stay tuned to our Community!

The post is synchronized to: FusionInsight Components

  • x
  • convention:

GADS
Created Mar 15, 2020 20:29:47

Thanks for sharing :)
View more
  • x
  • convention:

little_fish
Admin Created Mar 16, 2020 09:34:44

good
View more
  • x
  • convention:

hugu
Created Apr 11, 2022 13:54:00

Thanks for sharing
View more
  • x
  • convention:

bobi
Created Apr 12, 2022 06:11:51

Good share
View more
  • x
  • convention:

olive.zhao
olive.zhao Created Apr 12, 2022 06:52:53 (0) (0)
Thanks!  
NTan33
Created Apr 12, 2022 06:41:13

Quite a read indeed.
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.