|
jors
Created Aug 23, 2019 15:38:12
[Basic I/O switchover procedure after cables are disconnected]
After cables are disconnected, the HBA or iSCSI initiator detects the disconnection. I/O queues hang for some time and I/Os return to zero. An error code about device disconnection is returned. Multipathing software (MPIO/MSDSM or UltraPath) receives the returned I/Os and immediately starts a failover when it detects the error code. Services are restarted.
[Microsoft official description about two iSCSI initiator parameters]
Microsoft iSCSI Software Initiator Version 2.X Users Guide
MaxRequestHoldTime
Maximum time (in seconds) for which requests will be queued if connection to the target is lost and the connection is being retried. After this hold period, requests will be failed with "error no device" and device (disk) will be removed from the system.
The default value is 60 seconds
LinkDownTime
This value determines how long requests will be held in the device queue and retried if the connection to the target is lost. If MPIO is installed this value is used. If MPIO is not installed MaxRequestHoldTime is used instead.
The default value for this is 15 seconds.
After iSCSI detects the disconnection, I/Os hang for some time and return to zero. If MPIO is not installed, iSCSI uses MaxRequestHoldTime. If MPIO is installed, iSCSI uses LinkDownTime.
You can set the two parameters to control the time of a failover. Once the iSCSI imitator returns I/Os, the multipathing software performs a failover immediately after it receives the error code error no device. Little time is wasted in the multipathing software.
[Experience]
Based on previous experience, modifying two iSCSI imitator timeout parameters can control the time of a failover within 30s. However, the I/O switchover time may sometimes exceed 30s and may even reach 50 to 60s even tested repeatedly in the same environment.
Records of compatibility tests 08R2+HVSC99+ISCSI+10GE:
Run commands to modify MaxRequestHoldTime=5 and linkdowntime=15. Remove the cables and verify the modification. LUN failover starts. The time taken for I/Os to return to zero is about 33s.
Run commands to modify MaxRequestHoldTime=5 and linkdowntime=10. Remove the cables and verify the modification. LUN failover starts. The time taken for I/Os to return to zero is about 27s.
Run commands to modify MaxRequestHoldTime=5 and linkdowntime=5. Remove the cables and verify the modification. LUN failover starts. The time taken for I/Os to return to zero is about 23s.
Records of repeated tests in the same environment:
Run commands to modify MaxRequestHoldTime=11 and linkdowntime=12. Remove the cables and verify the modification. LUN failover starts. The time taken for I/Os to return to zero is about 30s. Occasionally, the time required is 60s.
Test results show that the I/O return-to-zero time equals to linkdowntime + linkdown detection time. In the records, the linkdown detection time stays at 17 to 18s. I/O return-to-zero time longer than 30s is caused by longer detection time.
The iSCSI initiator detects the following system log records after linkdown:
Error 10/23/2013 5:13:21 PM iScsiPrt 20 None Connection to the target was lost. The initiator will attempt to retry the connection.
In several tests, I/O hang time after linkdown is longer than the timeout parameter linkdowntime.
[Problem Analysis by Microsoft]
Microsoft iSCSI Software Initiator Version 2.X Users Guide
Switchover time involves two scenarios: 1. Cables of the hosts are removed. 2. Cables of the storage are removed (switches are included).
Conclusion:
Scenario 1: When cables are removed from hosts, a message about media disconnection is sent to network components and processed at the network layer. The processing is transparent to the storage stack.
Scenario 2: No mechanism is available to immediately detect that paths are unavailable if switch cables are removed. I/Os are processed as if the network connection is normal. The hosts can only detect the problem after I/Os time out. For a normal TCP network connection, the default data transmission retry time is 3. The initial retry takes three seconds. The retry time doubles after each try. Connections are reset after three retries and a linkdown message is sent to iSCSI. iSCSI starts to suspend the I/O queue until the timeout period is over.
In both scenarios, the iSCSI transmits data through the network layer. The disconnection detection time depends on where the connection is lost (physically) and what settings are configured for intermediate link transmission. The time varies depending on link speed, round trip time (RTT), and settings at the switch.
In both scenarios, because iSCSI uses the network layer as a transport, the connection failure detection time will depend on where the connection is lost (physically) and what settings are configured for retransmit behavior just as it would be for any network-level connection failure. This time can vary depending on link speed, RTT, settings at the switch, etc. so is always going to be variable in nature. From a TCP perspective, normal retransmit behavior is 3 seconds for the first, and a gap that doubles between each subsequent up to a maximum of 60 seconds. This behavior can vary however depending on whether it's a data retransmission or a SYN on a new connection.
[Conclusion]
For iSCSI scenarios:
Switchover time (time required for the iSCSI initiator to return I/Os) = Linkdown detection time + iSCSI timeout
The iSCSI transmits data through the network layer. The disconnection detection time depends on where the connection is lost (physically) and what settings are configured for intermediate link transmission. The time varies depending on link speed, round trip time (RTT), and settings at the switch.
Hosts can only change the I/O switchover time by modifying the iSCSI timeout parameter. Since linkdown detection time varies, the switchover time cannot be precisely controlled.
To solve this problem, configure the linkdown timeout parameter following chapter 7 Timeout Parameter Settings upon a Link Down Failure in the OceanStor UltraPath for Windows V100R008C50 User Guide 01.
|
|
|
|
|
|
