Got it

Heap Memory of an Elasticsearch Instance Is Insufficient Because the fielddata cache Occupies a Large Amount of Heap Memory

Latest reply: Nov 21, 2021 14:24:37 267 2 3 0 0

Hey there! 

This is a case about "Heap Memory of an Elasticsearch Instance Is Insufficient Because the fielddata cache Occupies a Large Amount of Heap Memory"

Applicable Version

6.5.x

Context and Symptom

The fielddata parameter of fields is enabled. As a result, the heap memory of the Elasticsearch instance is severely insufficient.

Cause Analysis

If fields of the text type are used for aggregation and sorting, you need to set fielddata of the fields to true. If fielddata is enabled, a large amount of heap memory is occupied. However, fielddata cache uses the LRU policy and is not recycled by garbage collection. If the fielddata cache usage threshold is not set, the fielddata cache increases until the heap memory of the entire Elasticsearch instance is used up.

Solution

1. On FusionInsight Manager, add the custom parameter indices.fielddata.cache.size to the Elasticsearch configuration to restrict the heap memory use of fielddata. For example, to restrict that fielddata cache can use 20% of heap memory at most, set indices.fielddata.cache.size to 20%.

2. Set the fielddata circuit breaker to limit the requests where fielddata cache uses too much heap memory, for example, the requests where the fielddata cache uses more than 40% of the heap memory.

curl -XPUT --negotiate -k -u : "https://ip:port/_cluster/settings" -H 'Content-Type: application/json' -d
'{
"persistent":{
"indices.breaker.fielddata.limit": "40%"
}
}'

3. Modify the service logic to avoid using fields of the text type for aggregation and sorting, reducing the heap memory use of fielddata cache.

4. Restart the Elasticsearch service to release the heap memory and make the custom parameter settings take effect.

Please feel free to ask us any questions!

  • x
  • convention:

olive.zhao
Admin Created Nov 16, 2021 09:56:22

Thanks for your sharing!
View more
  • x
  • convention:

user_4358465
Created Nov 21, 2021 14:24:37

Thanks for the provided solution
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.