The customer reports that log query is slow. Sometimes, it takes about 10 minutes to display the log query result, and the system does not immediately respond when they click to display the next page. However, the log query result can be displayed quickly sometimes.
The software version:
And the customer showed us the snapshot when he did the test.
|
(1) Checked the search condition, and found there is no special condition, no matter the period lasts for one day or several days, the issue will occur sometimes.
 (2) Simulated in Lab, and found there is same phenomenon that sometimes it takes about 10 minutes to display the log query result. After we analyzed the working implement about the search, we found that currently the situation is normal. The following is the analyze result: LogCenter logs are written into the database file in time sequence. When a user queries logs: (2.1) The system determines the files to be traversed based on the query time scope and then traverses the log files in time sequence (from the latest file to the earliest one). (2.2) The LogCenter does not immediately send obtained data to the analyzer. Instead, it sends data to the analyzer after it obtains 91 pieces of data. After the analyzer delivers a query command to the collector, the collector queries data based on the specified search criteria. After the first 91 pieces of data are sent to the analyzer, the homepage is displayed. Then, the collector sends 1024 pieces of data each time to the analyzer for the display of the second and later pages. (2.3) A page displays a maximum of 5000 pieces of data. (2.4) The LogCenter can traverse 5000 pieces of data per second. According to the experiment results in the lab, the LogCenter can traverse about 8000 pieces of data per second. The following figure shows the query speeds in the lab. You can see that one log is displayed every 30 seconds. (3) The LogCenter query speed is fixed. However, it seems that the query speed is not fixed because the search criteria are different. There are two examples: l It seems that the query speed is high: The logs matching the search criteria are in the latest log file. If there are more than 91 matching logs, the homepage is quickly displayed. l It seems that the query speed is low: There are less than 90 matching logs in all log files, or the logs are in early log files.
Root Cause
The analysis of logs obtained from the live network shows that the current query speed is reasonable and higher than that in the lab. The logs indicate that there are about 86,400,000 pieces of data generated every day on the network. If the query speed is 40,000 pieces per second, it takes about 36 minutes to traverse one-day data. The data query of the LogCenter is normal.
Solution
l Specify a short query time range to reduce the amount of data to be traversed. l Keep the firewall time and LogCenter server time consistent to prevent mismatch of search criteria due to the time inconsistency. l Use query tasks, which can run on the backend and can be executed concurrently without affecting other operations.
Suggestions
l Specify a short query time range to reduce the amount of data to be traversed. l Keep the firewall time and LogCenter server time consistent to prevent mismatch of search criteria due to the time inconsistency. l Use query tasks, which can run on the backend and can be executed concurrently without affecting other operations. |
|
Comment
