Collect logs of V3 storage

Created: Aug 16, 2019 10:26:20Latest reply: Aug 17, 2019 07:24:41 195 5 1 3
  Rewarded Hi-coins: 1 (problem resolved)

I want to collect the performance logs of 5800V3, is there any simple method? thanks very much.

  • x
  • convention:

Featured Answers
jors
Created Aug 16, 2019 11:32:34 Helpful(5) Helpful(5)

Hi there!

Collect logs of V3 storage-3036547-1
  • x
  • convention:

%E5%8E%9A%E7%A7%AF%E8%96%84%E5%8F%91
wissal
MVE Created Aug 16, 2019 11:12:17 Helpful(1) Helpful(1)

Hi!

Performance is associated with the entire process. A performance problem may occur in any aspect of the entire I/O path. For example, the host application may have an error, the network where the host resides may be congested, or the back-end storage system may have an error. The performance of a storage system is determined by the weakest point. Therefore, you must check all aspects to find out the performance bottleneck and eliminate it.
Context

A common method of locating a performance problem is to compare the average latency of a host with that of a storage system and then determine whether the problem resides on the host side, the network link layer, or the storage side.
  • If both the latency on the host side and that on the storage side are large and the difference between the latencies is small, the problem may reside in the storage system. Common reasons are as follows: The disk performance reaches the upper limit; the mirror bandwidth reaches the upper limit; a short-term performance deterioration occurs because LUN formatting is not complete.
    Collect logs of V3 storage-3036529-1 NOTE:
    The preceding problems are related to the read latency. Write latency includes the time spent in transmitting data from a host to a storage system. Therefore, if the write latency is large, the problem is not always caused by the storage system, and you should check all the aspects that may cause the problem.
  • If the latency on the host side is much larger than that on the storage side, the configuration on the host side may be inappropriate, leading to a performance or network link problem. Common reasons are as follows: I/Os are stacked because the concurrency capability of the block device or HBA is insufficient; the CPU usage of a host reaches 100%; the bandwidth reaches a bottleneck; the switch configuration is inappropriate; the multipathing software selects paths incorrectly.
  • After determining the location where a performance problem resides, such as on the host, storage, or network side, analyze and troubleshot the problem.
Procedure

  • Check the latency on the host side.
    • On a Linux host, use different tools to query the host latency.
      • Use the performance statistics function of service software, such as the AWR report function of Oracle, to query the host latency.
      • Use iostat, a Linux disk I/O query tool, to query the latency.Run iostat -kx 1.
        Collect logs of V3 storage-3036529-2

        In the command output, await indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
      • Use vdbench, a Linux performance test tool, to query the latency.
        Collect logs of V3 storage-3036529-3
        In the command output, resp indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
    • On a Windows host, use different tools to query the host latency.
      • Use the performance statistics function of service software to query the host latency.
      • Use IOmeter, a performance test tool commonly used in Windows, to query the host latency.Collect logs of V3 storage-3036529-4
      • Use the performance monitoring tool delivered with Windows to query the host latency. Windows Performance Monitor, a performance monitoring tool integrated with Windows, can monitor the performance of CPUs, memory, disks, network connections, and applications.
        The method of using Windows Performance Monitor to monitor disk performance is as follows:
        • On the Windows desktop, choose Start > Run. In the Run dialog box, enter perfmon to open the performance monitoring tool. The Performance Monitor window is displayed.
        • In the left navigation tree, choose Monitoring Tools > Performance Monitor and click Collect logs of V3 storage-3036529-5 to add performance items.
        • In the Add Counters window, select PhysicalDisk and add performance items that you want to monitor. Then, choose Add > OK. Windows Performance Monitor starts monitoring disk performance.
      • Table 4-1 describes performance items related to latency.
        Table 4-1  Disk performance items related to latency[tr]Indicator
        Subitem
        Description
        [/tr]
        Latency indicator
        Avg. Disk sec/Transfer
        Average time in processing each I/O on the storage side, expressed in milliseconds.
        Avg. Disk sec/Read
        Average time in processing each read I/O on the storage side.
        Avg. Disk sec/Write
        Average time in processing each write I/O on the storage side.


  • Check the latency on the storage side.

  • Use SystemReporter to query the latency on the storage side.Operation path: Monitoring > Real-Time Monitoring > Controller
    Collect logs of V3 storage-3036529-6
  • Run the CLI command to query the latency on the storage side.
    Log in to the CLI of the storage system and run show performance controller to query the average I/O response time of the specified controller, namely, Average I/O Latency (ms).admin:/>show performance controller controller_id=0A0.Max. Bandwidth (MB/s)                                 1.Usage Ratio (%)                                       2.Queue Length                                          3.Bandwidth (MB/s)                                      4.Throughput(IOPS) (IO/s)                               5.Read Bandwidth (MB/s)                                 6.Average Read I/O Size (KB)                            7.Read Throughput(IOPS) (IO/s)                          8.Write Bandwidth (MB/s)                                9.Average Write I/O Size (KB)                           10.Write Throughput(IOPS) (IO/s)                        11.Service Time (Excluding Queue Time) (ms)             12.Read I/O distribution: 512 B                         13.Read I/O distribution: 1 KB                          14.Read I/O distribution: 2 KB                          15.Read I/O distribution: 4 KB                          16.Read I/O distribution: 8 KB                          17.Read I/O distribution: 16 KB                         18.Read I/O distribution: 32 KB                         19.Read I/O distribution: 64 KB                         20.Read I/O distribution: 128 KB                        21.Read I/O distribution: 256 KB                        22.Read I/O distribution: 512 KB                        23.Write I/O distribution: 512 B                        24.Write I/O distribution: 1 KB                         25.Write I/O distribution: 2 KB                         26.Write I/O distribution: 4 KB                         27.Write I/O distribution: 8 KB                         28.Write I/O distribution: 16 KB                        29.Write I/O distribution: 32 KB                        30.Write I/O distribution: 64 KB                        31.Write I/O distribution: 128 KB                       32.Write I/O distribution: 256 KB                       33.Write I/O distribution: 512 KB                       34.CPU Usage (%)                                        35.Avg. Cache Usage (%)                                 36.Average I/O Latency (ms)                             37.Max. I/O Latency (ms)                                38.Percentage of Cache Flushes to Write Requests (%)    39.Cache Flushing Bandwidth (MB/s)                      40.Full-stripe Write Request                            41.Cache Read Usage (%)                                 42.Cache Write Usage (%)                                43.Average Read I/O Latency                             44.Average Write I/O Latency                            45.Average IO Size                                      46.Complete SCSI commands per second                    47.Verify commands per second                           48.% Read                                               49.% Write                                              50.Max IOPS(IO/s)                                       51.File Bandwidth(MB/s)                                 52.File OPS                                             53.File Bandwidth(MB/s) Input item(s) number seperated by comma:
    Collect logs of V3 storage-3036529-7 NOTE:
    For details about how to log in to the CLI of a storage system and the usage of CLI commands, see the Command Reference of the corresponding version.

For more details please refer to the link:

Thanks!
  • x
  • convention:

Telecommunications%20engineer%2C%20currently%20senior%20project%20manager%20at%20an%20operator%2C%20partner%20of%20Huawei%2C%20in%20the%20radio%20access%20network%20department%2C%20for%2020%20years%20I%20managed%20several%20types%20of%20projects%2C%20for%20the%20different%20nodes%20of%20the%20network.
All Answers
wissal
wissal MVE Created Aug 16, 2019 11:12:17 Helpful(1) Helpful(1)

Hi!

Performance is associated with the entire process. A performance problem may occur in any aspect of the entire I/O path. For example, the host application may have an error, the network where the host resides may be congested, or the back-end storage system may have an error. The performance of a storage system is determined by the weakest point. Therefore, you must check all aspects to find out the performance bottleneck and eliminate it.
Context

A common method of locating a performance problem is to compare the average latency of a host with that of a storage system and then determine whether the problem resides on the host side, the network link layer, or the storage side.
  • If both the latency on the host side and that on the storage side are large and the difference between the latencies is small, the problem may reside in the storage system. Common reasons are as follows: The disk performance reaches the upper limit; the mirror bandwidth reaches the upper limit; a short-term performance deterioration occurs because LUN formatting is not complete.
    Collect logs of V3 storage-3036529-1 NOTE:
    The preceding problems are related to the read latency. Write latency includes the time spent in transmitting data from a host to a storage system. Therefore, if the write latency is large, the problem is not always caused by the storage system, and you should check all the aspects that may cause the problem.
  • If the latency on the host side is much larger than that on the storage side, the configuration on the host side may be inappropriate, leading to a performance or network link problem. Common reasons are as follows: I/Os are stacked because the concurrency capability of the block device or HBA is insufficient; the CPU usage of a host reaches 100%; the bandwidth reaches a bottleneck; the switch configuration is inappropriate; the multipathing software selects paths incorrectly.
  • After determining the location where a performance problem resides, such as on the host, storage, or network side, analyze and troubleshot the problem.
Procedure

  • Check the latency on the host side.
    • On a Linux host, use different tools to query the host latency.
      • Use the performance statistics function of service software, such as the AWR report function of Oracle, to query the host latency.
      • Use iostat, a Linux disk I/O query tool, to query the latency.Run iostat -kx 1.
        Collect logs of V3 storage-3036529-2

        In the command output, await indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
      • Use vdbench, a Linux performance test tool, to query the latency.
        Collect logs of V3 storage-3036529-3
        In the command output, resp indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
    • On a Windows host, use different tools to query the host latency.
      • Use the performance statistics function of service software to query the host latency.
      • Use IOmeter, a performance test tool commonly used in Windows, to query the host latency.Collect logs of V3 storage-3036529-4
      • Use the performance monitoring tool delivered with Windows to query the host latency. Windows Performance Monitor, a performance monitoring tool integrated with Windows, can monitor the performance of CPUs, memory, disks, network connections, and applications.
        The method of using Windows Performance Monitor to monitor disk performance is as follows:
        • On the Windows desktop, choose Start > Run. In the Run dialog box, enter perfmon to open the performance monitoring tool. The Performance Monitor window is displayed.
        • In the left navigation tree, choose Monitoring Tools > Performance Monitor and click Collect logs of V3 storage-3036529-5 to add performance items.
        • In the Add Counters window, select PhysicalDisk and add performance items that you want to monitor. Then, choose Add > OK. Windows Performance Monitor starts monitoring disk performance.
      • Table 4-1 describes performance items related to latency.
        Table 4-1  Disk performance items related to latency[tr]Indicator
        Subitem
        Description
        [/tr]
        Latency indicator
        Avg. Disk sec/Transfer
        Average time in processing each I/O on the storage side, expressed in milliseconds.
        Avg. Disk sec/Read
        Average time in processing each read I/O on the storage side.
        Avg. Disk sec/Write
        Average time in processing each write I/O on the storage side.


  • Check the latency on the storage side.

  • Use SystemReporter to query the latency on the storage side.Operation path: Monitoring > Real-Time Monitoring > Controller
    Collect logs of V3 storage-3036529-6
  • Run the CLI command to query the latency on the storage side.
    Log in to the CLI of the storage system and run show performance controller to query the average I/O response time of the specified controller, namely, Average I/O Latency (ms).admin:/>show performance controller controller_id=0A0.Max. Bandwidth (MB/s)                                 1.Usage Ratio (%)                                       2.Queue Length                                          3.Bandwidth (MB/s)                                      4.Throughput(IOPS) (IO/s)                               5.Read Bandwidth (MB/s)                                 6.Average Read I/O Size (KB)                            7.Read Throughput(IOPS) (IO/s)                          8.Write Bandwidth (MB/s)                                9.Average Write I/O Size (KB)                           10.Write Throughput(IOPS) (IO/s)                        11.Service Time (Excluding Queue Time) (ms)             12.Read I/O distribution: 512 B                         13.Read I/O distribution: 1 KB                          14.Read I/O distribution: 2 KB                          15.Read I/O distribution: 4 KB                          16.Read I/O distribution: 8 KB                          17.Read I/O distribution: 16 KB                         18.Read I/O distribution: 32 KB                         19.Read I/O distribution: 64 KB                         20.Read I/O distribution: 128 KB                        21.Read I/O distribution: 256 KB                        22.Read I/O distribution: 512 KB                        23.Write I/O distribution: 512 B                        24.Write I/O distribution: 1 KB                         25.Write I/O distribution: 2 KB                         26.Write I/O distribution: 4 KB                         27.Write I/O distribution: 8 KB                         28.Write I/O distribution: 16 KB                        29.Write I/O distribution: 32 KB                        30.Write I/O distribution: 64 KB                        31.Write I/O distribution: 128 KB                       32.Write I/O distribution: 256 KB                       33.Write I/O distribution: 512 KB                       34.CPU Usage (%)                                        35.Avg. Cache Usage (%)                                 36.Average I/O Latency (ms)                             37.Max. I/O Latency (ms)                                38.Percentage of Cache Flushes to Write Requests (%)    39.Cache Flushing Bandwidth (MB/s)                      40.Full-stripe Write Request                            41.Cache Read Usage (%)                                 42.Cache Write Usage (%)                                43.Average Read I/O Latency                             44.Average Write I/O Latency                            45.Average IO Size                                      46.Complete SCSI commands per second                    47.Verify commands per second                           48.% Read                                               49.% Write                                              50.Max IOPS(IO/s)                                       51.File Bandwidth(MB/s)                                 52.File OPS                                             53.File Bandwidth(MB/s) Input item(s) number seperated by comma:
    Collect logs of V3 storage-3036529-7 NOTE:
    For details about how to log in to the CLI of a storage system and the usage of CLI commands, see the Command Reference of the corresponding version.

For more details please refer to the link:

Thanks!
  • x
  • convention:

Telecommunications%20engineer%2C%20currently%20senior%20project%20manager%20at%20an%20operator%2C%20partner%20of%20Huawei%2C%20in%20the%20radio%20access%20network%20department%2C%20for%2020%20years%20I%20managed%20several%20types%20of%20projects%2C%20for%20the%20different%20nodes%20of%20the%20network.
jors
jors Created Aug 16, 2019 11:28:02 Helpful(0) Helpful(0)

Saonian, get, no thanks!!!!

  • x
  • convention:

%E5%8E%9A%E7%A7%AF%E8%96%84%E5%8F%91
jors
jors Created Aug 16, 2019 11:32:34 Helpful(5) Helpful(5)

Hi there!

Collect logs of V3 storage-3036547-1
  • x
  • convention:

%E5%8E%9A%E7%A7%AF%E8%96%84%E5%8F%91
Spensor
Spensor Created Aug 16, 2019 15:58:09 Helpful(0) Helpful(0)

helpful
  • x
  • convention:

Junmin
Junmin Created Aug 17, 2019 07:24:41 Helpful(0) Helpful(0)

For V3 and dorado V3 storage, if the storage version is not that old, for example V300R006C20/V5R7C10 or dorado V3R1C21SPC100 or later, you can use SmartKit to collect storage perforamance logs.
Just like this.
20190817-102401(eSpace)

20190817-102408(eSpace)20190817-102412(eSpace)

This article contains more resources

You need to log in to download or view. No account?Register

x
  • x
  • convention:

Reply

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login