Collect logs of V3 storage

jors · Created Aug 16, 2019 11:32:34

Hi there!

wissal · Created Aug 16, 2019 11:12:17

Hi!

Performance is associated with the entire process. A performance problem may occur in any aspect of the entire I/O path. For example, the host application may have an error, the network where the host resides may be congested, or the back-end storage system may have an error. The performance of a storage system is determined by the weakest point. Therefore, you must check all aspects to find out the performance bottleneck and eliminate it.

Context

A common method of locating a performance problem is to compare the average latency of a host with that of a storage system and then determine whether the problem resides on the host side, the network link layer, or the storage side.

If both the latency on the host side and that on the storage side are large and the difference between the latencies is small, the problem may reside in the storage system. Common reasons are as follows: The disk performance reaches the upper limit; the mirror bandwidth reaches the upper limit; a short-term performance deterioration occurs because LUN formatting is not complete.
NOTE:
The preceding problems are related to the read latency. Write latency includes the time spent in transmitting data from a host to a storage system. Therefore, if the write latency is large, the problem is not always caused by the storage system, and you should check all the aspects that may cause the problem.
If the latency on the host side is much larger than that on the storage side, the configuration on the host side may be inappropriate, leading to a performance or network link problem. Common reasons are as follows: I/Os are stacked because the concurrency capability of the block device or HBA is insufficient; the CPU usage of a host reaches 100%; the bandwidth reaches a bottleneck; the switch configuration is inappropriate; the multipathing software selects paths incorrectly.
After determining the location where a performance problem resides, such as on the host, storage, or network side, analyze and troubleshot the problem.

Procedure

Check the latency on the host side.

On a Linux host, use different tools to query the host latency.
- Use the performance statistics function of service software, such as the AWR report function of Oracle, to query the host latency.
- Use iostat, a Linux disk I/O query tool, to query the latency.Run iostat -kx 1.
  
  In the command output, await indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
- Use vdbench, a Linux performance test tool, to query the latency.
  
  In the command output, resp indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.

On a Windows host, use different tools to query the host latency.

Use the performance statistics function of service software to query the host latency.
Use IOmeter, a performance test tool commonly used in Windows, to query the host latency.
Use the performance monitoring tool delivered with Windows to query the host latency. Windows Performance Monitor, a performance monitoring tool integrated with Windows, can monitor the performance of CPUs, memory, disks, network connections, and applications.
The method of using Windows Performance Monitor to monitor disk performance is as follows:
- On the Windows desktop, choose Start > Run. In the Run dialog box, enter perfmon to open the performance monitoring tool. The Performance Monitor window is displayed.
- In the left navigation tree, choose Monitoring Tools > Performance Monitor and click to add performance items.
- In the Add Counters window, select PhysicalDisk and add performance items that you want to monitor. Then, choose Add > OK. Windows Performance Monitor starts monitoring disk performance.

Table 4-1 describes performance items related to latency.

Table 4-1 Disk performance items related to latency[tr]Indicator
Subitem
Description
[/tr]

Latency indicator	Avg. Disk sec/Transfer	Average time in processing each I/O on the storage side, expressed in milliseconds.
	Avg. Disk sec/Read	Average time in processing each read I/O on the storage side.
	Avg. Disk sec/Write	Average time in processing each write I/O on the storage side.

Check the latency on the storage side.

Use SystemReporter to query the latency on the storage side.Operation path: Monitoring > Real-Time Monitoring > Controller
Run the CLI command to query the latency on the storage side.
Log in to the CLI of the storage system and run show performance controller to query the average I/O response time of the specified controller, namely, Average I/O Latency (ms).admin:/>show performance controller controller_id=0A0.Max. Bandwidth (MB/s) 1.Usage Ratio (%) 2.Queue Length 3.Bandwidth (MB/s) 4.Throughput(IOPS) (IO/s) 5.Read Bandwidth (MB/s) 6.Average Read I/O Size (KB) 7.Read Throughput(IOPS) (IO/s) 8.Write Bandwidth (MB/s) 9.Average Write I/O Size (KB) 10.Write Throughput(IOPS) (IO/s) 11.Service Time (Excluding Queue Time) (ms) 12.Read I/O distribution: 512 B 13.Read I/O distribution: 1 KB 14.Read I/O distribution: 2 KB 15.Read I/O distribution: 4 KB 16.Read I/O distribution: 8 KB 17.Read I/O distribution: 16 KB 18.Read I/O distribution: 32 KB 19.Read I/O distribution: 64 KB 20.Read I/O distribution: 128 KB 21.Read I/O distribution: 256 KB 22.Read I/O distribution: 512 KB 23.Write I/O distribution: 512 B 24.Write I/O distribution: 1 KB 25.Write I/O distribution: 2 KB 26.Write I/O distribution: 4 KB 27.Write I/O distribution: 8 KB 28.Write I/O distribution: 16 KB 29.Write I/O distribution: 32 KB 30.Write I/O distribution: 64 KB 31.Write I/O distribution: 128 KB 32.Write I/O distribution: 256 KB 33.Write I/O distribution: 512 KB 34.CPU Usage (%) 35.Avg. Cache Usage (%) 36.Average I/O Latency (ms) 37.Max. I/O Latency (ms) 38.Percentage of Cache Flushes to Write Requests (%) 39.Cache Flushing Bandwidth (MB/s) 40.Full-stripe Write Request 41.Cache Read Usage (%) 42.Cache Write Usage (%) 43.Average Read I/O Latency 44.Average Write I/O Latency 45.Average IO Size 46.Complete SCSI commands per second 47.Verify commands per second 48.% Read 49.% Write 50.Max IOPS(IO/s) 51.File Bandwidth(MB/s) 52.File OPS 53.File Bandwidth(MB/s) Input item(s) number seperated by comma:
NOTE:
For details about how to log in to the CLI of a storage system and the usage of CLI commands, see the Command Reference of the corresponding version.

For more details please refer to the link:

https://support.huawei.com/enterprise/en/doc/EDOC1000088584?section=j007

Thanks!

wissal · Created Aug 16, 2019 11:12:17

Hi!

Performance is associated with the entire process. A performance problem may occur in any aspect of the entire I/O path. For example, the host application may have an error, the network where the host resides may be congested, or the back-end storage system may have an error. The performance of a storage system is determined by the weakest point. Therefore, you must check all aspects to find out the performance bottleneck and eliminate it.

Context

A common method of locating a performance problem is to compare the average latency of a host with that of a storage system and then determine whether the problem resides on the host side, the network link layer, or the storage side.

If both the latency on the host side and that on the storage side are large and the difference between the latencies is small, the problem may reside in the storage system. Common reasons are as follows: The disk performance reaches the upper limit; the mirror bandwidth reaches the upper limit; a short-term performance deterioration occurs because LUN formatting is not complete.
NOTE:
The preceding problems are related to the read latency. Write latency includes the time spent in transmitting data from a host to a storage system. Therefore, if the write latency is large, the problem is not always caused by the storage system, and you should check all the aspects that may cause the problem.
If the latency on the host side is much larger than that on the storage side, the configuration on the host side may be inappropriate, leading to a performance or network link problem. Common reasons are as follows: I/Os are stacked because the concurrency capability of the block device or HBA is insufficient; the CPU usage of a host reaches 100%; the bandwidth reaches a bottleneck; the switch configuration is inappropriate; the multipathing software selects paths incorrectly.
After determining the location where a performance problem resides, such as on the host, storage, or network side, analyze and troubleshot the problem.

Procedure

Check the latency on the host side.

On a Linux host, use different tools to query the host latency.
- Use the performance statistics function of service software, such as the AWR report function of Oracle, to query the host latency.
- Use iostat, a Linux disk I/O query tool, to query the latency.Run iostat -kx 1.
  
  In the command output, await indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.
- Use vdbench, a Linux performance test tool, to query the latency.
  
  In the command output, resp indicates the average time in processing each I/O request, namely, the I/O response time, expressed in milliseconds.

On a Windows host, use different tools to query the host latency.

Use the performance statistics function of service software to query the host latency.
Use IOmeter, a performance test tool commonly used in Windows, to query the host latency.
Use the performance monitoring tool delivered with Windows to query the host latency. Windows Performance Monitor, a performance monitoring tool integrated with Windows, can monitor the performance of CPUs, memory, disks, network connections, and applications.
The method of using Windows Performance Monitor to monitor disk performance is as follows:
- On the Windows desktop, choose Start > Run. In the Run dialog box, enter perfmon to open the performance monitoring tool. The Performance Monitor window is displayed.
- In the left navigation tree, choose Monitoring Tools > Performance Monitor and click to add performance items.
- In the Add Counters window, select PhysicalDisk and add performance items that you want to monitor. Then, choose Add > OK. Windows Performance Monitor starts monitoring disk performance.

Table 4-1 describes performance items related to latency.

Table 4-1 Disk performance items related to latency[tr]Indicator
Subitem
Description
[/tr]

Latency indicator	Avg. Disk sec/Transfer	Average time in processing each I/O on the storage side, expressed in milliseconds.
	Avg. Disk sec/Read	Average time in processing each read I/O on the storage side.
	Avg. Disk sec/Write	Average time in processing each write I/O on the storage side.

Check the latency on the storage side.

Use SystemReporter to query the latency on the storage side.Operation path: Monitoring > Real-Time Monitoring > Controller
Run the CLI command to query the latency on the storage side.
Log in to the CLI of the storage system and run show performance controller to query the average I/O response time of the specified controller, namely, Average I/O Latency (ms).admin:/>show performance controller controller_id=0A0.Max. Bandwidth (MB/s) 1.Usage Ratio (%) 2.Queue Length 3.Bandwidth (MB/s) 4.Throughput(IOPS) (IO/s) 5.Read Bandwidth (MB/s) 6.Average Read I/O Size (KB) 7.Read Throughput(IOPS) (IO/s) 8.Write Bandwidth (MB/s) 9.Average Write I/O Size (KB) 10.Write Throughput(IOPS) (IO/s) 11.Service Time (Excluding Queue Time) (ms) 12.Read I/O distribution: 512 B 13.Read I/O distribution: 1 KB 14.Read I/O distribution: 2 KB 15.Read I/O distribution: 4 KB 16.Read I/O distribution: 8 KB 17.Read I/O distribution: 16 KB 18.Read I/O distribution: 32 KB 19.Read I/O distribution: 64 KB 20.Read I/O distribution: 128 KB 21.Read I/O distribution: 256 KB 22.Read I/O distribution: 512 KB 23.Write I/O distribution: 512 B 24.Write I/O distribution: 1 KB 25.Write I/O distribution: 2 KB 26.Write I/O distribution: 4 KB 27.Write I/O distribution: 8 KB 28.Write I/O distribution: 16 KB 29.Write I/O distribution: 32 KB 30.Write I/O distribution: 64 KB 31.Write I/O distribution: 128 KB 32.Write I/O distribution: 256 KB 33.Write I/O distribution: 512 KB 34.CPU Usage (%) 35.Avg. Cache Usage (%) 36.Average I/O Latency (ms) 37.Max. I/O Latency (ms) 38.Percentage of Cache Flushes to Write Requests (%) 39.Cache Flushing Bandwidth (MB/s) 40.Full-stripe Write Request 41.Cache Read Usage (%) 42.Cache Write Usage (%) 43.Average Read I/O Latency 44.Average Write I/O Latency 45.Average IO Size 46.Complete SCSI commands per second 47.Verify commands per second 48.% Read 49.% Write 50.Max IOPS(IO/s) 51.File Bandwidth(MB/s) 52.File OPS 53.File Bandwidth(MB/s) Input item(s) number seperated by comma:
NOTE:
For details about how to log in to the CLI of a storage system and the usage of CLI commands, see the Command Reference of the corresponding version.

For more details please refer to the link:

https://support.huawei.com/enterprise/en/doc/EDOC1000088584?section=j007

Thanks!

jors · Created Aug 16, 2019 11:28:02

Saonian, get, no thanks!!!!

jors · Created Aug 16, 2019 11:32:34

Hi there!

Spensor · Created Aug 16, 2019 15:58:09

helpful

Junmin · Created Aug 17, 2019 07:24:41

For V3 and dorado V3 storage, if the storage version is not that old, for example V300R006C20/V5R7C10 or dorado V3R1C21SPC100 or later, you can use SmartKit to collect storage perforamance logs.
Just like this.

This article contains more resources

You need to log in to download or view. No account? Register

x

Collect logs of V3 storage

Best answer

Recommended answer

This article contains more resources

Third Party’s Trade Secret

My Followers

Enterprise

Consumer

Corporate

Carriers

Collect logs of V3 storage

Best answer

Recommended answer

This article contains more resources

Third Party’s Trade Secret

My Followers