Got it

Slow log in elasticsearch

Latest reply: Jan 19, 2022 10:51:46 375 2 2 0 0

Hello all, 

This case mainly talks about "Slow log in elasticsearch"

Search Slow Log

Shard level slow search log allows to log slow search (query and fetch phases) into a dedicated log file.

Thresholds can be set for both the query phase of the execution, and the fetch phase, here is a sample:

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.search.slowlog.level: info

All of the above settings are dynamic and can be set for each index using the update indices settings API. For example:

PUT /my-index-000001/_settings{
 "index.search.slowlog.threshold.query.warn": "10s",
 "index.search.slowlog.threshold.query.info": "5s",
 "index.search.slowlog.threshold.query.debug": "2s",
 "index.search.slowlog.threshold.query.trace": "500ms",
 "index.search.slowlog.threshold.fetch.warn": "1s",
 "index.search.slowlog.threshold.fetch.info": "800ms",
 "index.search.slowlog.threshold.fetch.debug": "500ms",
 "index.search.slowlog.threshold.fetch.trace": "200ms",
 "index.search.slowlog.level": "info"}


By default, none are enabled (set to -1). Levels (warn, info, debug, trace) allow controlling under which logging level the log will be logged. Not all are required to be configured (for example, only warn threshold can be set). The benefit of several levels is the ability to quickly "grep" for specific thresholds breached.

The logging is done on the shard level scope, meaning the execution of a search request within a specific shard. It does not encompass the whole search request, which can be broadcast to several shards in order to execute. Some of the benefits of shard level logging are the association of the actual execution on the specific machine, compared with request level.

The logging file is configured by default using the following configuration (found in log4j2.properties):

appender.index_search_slowlog_rolling.type = RollingFileappender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
appender.index_search_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog.log
appender.index_search_slowlog_rolling.layout.type = PatternLayoutappender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %.-10000m%n
appender.index_search_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog-%i.log.gz
appender.index_search_slowlog_rolling.policies.type = Policiesappender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicyappender.index_search_slowlog_rolling.policies.size.size = 1GBappender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategyappender.index_search_slowlog_rolling.strategy.max = 4logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = trace
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
logger.index_search_slowlog_rolling.additivity = false

Identifying search slow log origin

It is often useful to identify what triggered a slow running query. If a call was initiated with an X-Opaque-ID header, then the user ID is included in Search Slow logs as an additional id field (scroll to the right).

[2030-08-30T11:59:37,786][WARN ][i.s.s.query              ] [node-0] [index6][0] took[78.4micros], took_millis[0], total_hits[0 hits], stats[], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"query":{"match_all":{"boost":1.0}}}], id[MY_USER_ID],

The user ID is also included in JSON logs.


{
 "type": "index_search_slowlog",
 "timestamp": "2030-08-30T11:59:37,786+02:00",
 "level": "WARN",
 "component": "i.s.s.query",
 "cluster.name": "distribution_run",
 "node.name": "node-0",
 "message": "[index6][0]",
 "took": "78.4micros",
 "took_millis": "0",
 "total_hits": "0 hits",
 "stats": "[]",
 "search_type": "QUERY_THEN_FETCH",
 "total_shards": "1",
 "source": "{\"query\":{\"match_all\":{\"boost\":1.0}}}",
 "id": "MY_USER_ID",
 "cluster.uuid": "Aq-c-PAeQiK3tfBYtig9Bw",
 "node.id": "D7fUYfnfTLa2D7y-xw6tZg"}

Index Slow log

The indexing slow log, similar in functionality to the search slow log. The log file name ends with _index_indexing_slowlog.log. Log and the thresholds are configured in the same way as the search slowlog. Index slowlog sample:

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms
index.indexing.slowlog.level: info
index.indexing.slowlog.source: 1000

All of the above settings are dynamic and can be set for each index using the update indices settings API. For example:

PUT /my-index-000001/_settings{
 "index.indexing.slowlog.threshold.index.warn": "10s",
 "index.indexing.slowlog.threshold.index.info": "5s",
 "index.indexing.slowlog.threshold.index.debug": "2s",
 "index.indexing.slowlog.threshold.index.trace": "500ms",
 "index.indexing.slowlog.level": "info",
 "index.indexing.slowlog.source": "1000"}

By default Elasticsearch will log the first 1000 characters of the _source in the slowlog. You can change that with index.indexing.slowlog.source. Setting it to false or will skip logging the source entirely and setting it to true will log the entire source regardless of size. The original _source is reformatted by default to make sure that it fits on a single logline. If preserving the original document format is important, you can turn off reformatting by setting index.indexing.slowlog.reformat to false, which will cause the source to be logged "as is" and can potentially span multiple log lines.

The index slow log file is configured by default in the log4j2.properties file:

appender.index_indexing_slowlog_rolling.type = RollingFileappender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
appender.index_indexing_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog.log
appender.index_indexing_slowlog_rolling.layout.type = PatternLayoutappender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %.-10000m%n
appender.index_indexing_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog-%i.log.gz
appender.index_indexing_slowlog_rolling.policies.type = Policiesappender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicyappender.index_indexing_slowlog_rolling.policies.size.size = 1GBappender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategyappender.index_indexing_slowlog_rolling.strategy.max = 4logger.index_indexing_slowlog.name = index.indexing.slowlog.index
logger.index_indexing_slowlog.level = trace
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
logger.index_indexing_slowlog.additivity = false

That's all, thanks!

  • x
  • convention:

olive.zhao
Admin Created Jan 14, 2022 02:10:23

Thanks for your sharing!
View more
  • x
  • convention:

user_4358465
Created Jan 19, 2022 10:51:46

Thanks, its very helpful
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.