How to Collect FusionCompute Dump Logs? Highlighted

Latest reply: Aug 9, 2019 10:02:10 203 2 8 1

【Issue Description】


During problem analysis, R&D engineers need to collect related dump logs to find out the root cause of the problem by analyzing logs. However, frontline engineers do not know the log structure and how to obtain dump logs.

【Keywords】Log path description, log collection

【Solution】FusionCompute log paths and log collection methods are provided to resolve this problem.

1Collecting Logs Based on FC+XEN


1.1    Collect VRM dump logs

(1) Use PuTTY tool to log in to the master VRM server by using the management IP address, and as the Gandalf user.

(2) Run the su command and enter the password of user as prompted to switch to user root.

(3) Run the TMOUT=0 command to prevent PuTTY from exiting due to timeout.

153501hzp0x2av2p024hhm.png?image.png.

(4)Run the cp -r /var/backuplog/galaxenginelog/20150424(Indicates the date)/ /home/GalaX8800/ command.

(5)Run the chown -R gandalf:GalaX8800 /home/GalaX8800/20150424 command.

(6) Log in to the master VRM VM as user gandalf by using winscp tool, copy the /home/GalaX8800/20150424/ directory, and pack it.

(7) After the copy is complete, run the rm -r /home/GalaX8800/20150424/ command to delete the directory.

(8) Send back the logs.


The method of searching for messages logs in the VRM dump logs is as follows:


Method I: In XEN, the messages logs of the VRM are dumped by size. So the message logs are not recorded in the VRM dump logs at each time. To find messages logs, perform the following steps

1. Open the current vim /var/log/messages log, press gg to go to the first line, and confirm the earliest time of the current messages log, for example, 9:00.

153602noqsvwshqsx8skyy.png?image.png

2. Then the message log was dumped last time should be in the package at 8: 00. You can run the unzip -l 20190612-08.zip | grep messages command to check whether the message log is in the package.

153620omywkm3ymlwotmfl.png?image.png

3. If you need to find the message logs at earlier time, you can use the preceding 1+2 method to cycle to get the message log, but you need to open the found message dump package to confirm the log dump time.


Method II

As shown in the following figure, in the dump directory of a certain date of VRM, run the ls | xargs -i unzip -l {} | grep messages

 command to check whether the zip package contains the messages log:

153652a66chz53xc6bth53.png?image.png

1.2 Method for CNA dump log collection:

The path for storing dumped logs on the server is as follows: /var/log/ [Host name]

In the dump logs under this directory, each directory represents all dump logs of a day. The dump logs of each day contain 24 directories. Each directory represents the dumped logs within an hour. The dumped logs in each hour contain four dump log files, the file names are log0.tar.gz, log1.tar.gz, log2.tar.gz, and log3.tar.gz, which respectively indicate the 45-60 minutes in the last hour and the logs dumped in the current hour 0-15 minutes, 15-30 minutes, and 30-45 minutes.

153726u5p5cdxc4x00f741.png?image.png

 (1) Use PuTTY to log in to the host. (As user "gandalf", and use the management IP address to log in to the host.)

(2) Run the su command and enter the password of the user “root” as prompted to switch to "root".

(3) Run the TMOUT=0 command to prevent the "PuTTY" from exiting due to timeout.

(4) Run the (cp -r /var/log/CNA01/03/ /home/GalaX8800/) command. Note that CNA01 is the host name. The host name varies according to the host. 03 indicates the third day of the current month.

(5)Run the (chown -R gandalf:GalaX8800 /home/GalaX8800) command.

(6) Log in to the host as user “Gandalf” by using winscp. After copying the /home/GalaX8800/03/ directory, delete the new directory (rm -r /home/GalaX8800/03/) from GalaX8800.

1.3 Method for the Kdump Logs Collection of the CNA Node : 

(1) Log in to the CNA node as root user by using PuTTY.

(2) Obtain the Kdump logs generated when the fault occurs from the following path on the CNA node: vmcore.gz and vmcore_panic_log.

/tmp/disk/tmp/ [System Breakdown Time] /

153820ik99qq65zyayqp3p.png?image.png

Note:

The Kdump log files are stored in a folder named in the format of YYYY-MM-DD-HH-MM-hh: Mm: SS – Time zone. The time is corresponding to the time of the time zone.

· vmcore.gz stores memory information.

· The vmcore_panic_log file saves the serial port exception information captured when the host is faulty.

1.4 Method for Logs Collection of the Serial Port on the CNA Node

Note: The methods for collecting serial port logs of the HUAWEI V1 server and V2 server are different. For heterogeneous servers, refer to the documents of the heterogeneous servers for collection.

  HUAWEI V1 Server


The serial port server needs to be deployed on the HUAWEI V1 server to collect serial port logs. The procedure is as follows (taking the CNA01 node as an example):

Run the following command on the CNA01 node to check the BMC IP address of the CNA01 node:

153857fugnxou5n83xw3r9.png?image.png

Select a node that can communicate with the CNA01 node as the serial port server. The serial port server cannot be the CNA01 itself.

ipmitool -I lanplus -H 130.10.4.72 -U root -P root sol activate | tee -a /home/serlog/CNA01.log

Log in to the serial port server and run the following command to specify the directory for saving the serial port information and the name of the serial port file:

ipmitool -I lanplus -H 130.10.4.72 -U root -P root sol activate | tee -a /home/serlog/CNA01.log


Note:

In the command, “130.10.4.72 -U root -P root” indicates the BMC IP address, user name, and password. Save the serial port log information as the CNA01.log file under the /home/serlog/ directory.

Check whether the serial port function is normal.

Use PuTTY to log in to CNA01 and run the following command:

echo hello > /dev/xvc0

Run the following command on the serial port server to check whether a new line "hello" is added:

cat /home/serlog/CNA01.log

* Yes, indicating that the serial port server works properly.

* No, the serial port server is abnormal. Go to step 1.

153956s7z6syi9b6tbtbnh.png?image.png

  HUAWEI V2 Server


The HUAWEI V2 server has a fixed storage for 1 MB serial port logs. To collect the logs, perform the following steps:

Use PuTTY to log in to the BMC and run the following command to collect serial port logs. /dev/shm/share/sol.dat is the serial port log.

154014u7ctxn27uo6z7hyh.png?image.png

Collection Method of logs based on the FC+KVM  

2.1 VRM log path and collection method: 

Note: VRM logs are backed up every one hour. Run the date command on the vrm server to check whether the vrm time is the same as the current time. If not, calculate the error time based on the time difference. For example, if the current time is 22:00 and the time on vrm is 14:00,then your operation at 22:00 needs to query logs at 14:00 on the VRM. If the corresponding log cannot be found on the VRM server, an active/standby switchover may occur. In this case, log in to the standby VRM node and search for related logs.

Current VRM log path:/var/log/galaxenginelog/vrm


154038fko0zssrzig9ts4s.png?image.png

History VRM log path:/var/backuplog/galaxenginelog/{date}/{hour}

154047pygdn1mawna25mby.png?image.png

For details about how to collect logs, see section 1.1 VRM log path and collection method.

2.2  CNA log path and collection method:  

    

Note:CNA logs are backed up every one hour. Run the date command on the CNA to check whether the time of the CNA is the same as the current time. If not, calculate the error time based on the time difference. For example, if the current time is 22:00 and the time on the CNA is 14:00, then your operation at 22:00 needs to query logs at 14:00 on the VRM.

Current CNA log path:/var/log/galaxenginelog/vna-api


154140jz1rfrk4z4kz4lz7.png?image.png

History CNA log path:/var/backuplog/galaxenginelog/{Date}/{hour}/vna-api-history

154151isphhihucekd5kim.png?image.png

To view CNA logs, perform the following steps:

You only need to pay attention to vna-server.log and vna-worker.log for the key information logs of the CNA.

1. vim Enter the vna-server.log log.

2. Search for the request id of the corresponding error information, for example, /76b7bedc-733c-4554-924f-537455ac8cb7.

3. Check whether the vna-server.log file contains error information.

4. If the vna-server.log file does not contain error information, check whether error information exists in the vna-worker.log file based on request id.

  • x
  • convention:

stephen.xu
Admin Created Jul 31, 2019 00:52:42 Helpful(0) Helpful(0)

It is useful for me to collect logs.
  • x
  • convention:

songminwang
Admin Created Aug 9, 2019 10:02:10 Helpful(0) Helpful(0)

Very help for meHow to Collect FusionCompute Dump Logs?-3029789-1
  • x
  • convention:

Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login