Hi team, here' s a new case.
[Problem Description]
For OptiX OSN 6800/6900 V3 V300R003C00 and later versions,
if a spare controller is replaced during the upgrade,
the T4 chip interface card on the controller fails to be upgraded.
The T4 chip interface card includes TOE optical interface card, TOE electrical interface card, and FCoE card.
[Fault Symptom]
The T4 chip interface card fails to be upgraded.
[Principle/Root Cause Analysis]
T4 chip interface card upgrade failure of OptiX OSN 6800/6900 V3
1. During the upgrade, a spare controller is replaced. After the software version of the spare controller is synchronized, the new software version matches the old logic (CPLD) version in a certain time window. As a result, the PCIe slot initialization fails.
2. After the faulty controller is replaced during the upgrade, if the T4 chip interface card is available, (including TOE optical interface cards, TOE electrical interface cards, and FCoE cards) After the T4 chip interface card is upgraded, the interface card is repaired so that the new firmware version of the interface card takes effect. Then, the version is verified to determine whether the upgrade is successful.
3. To make the upgrade take effect, the interface card repair process is required. However, the repair process checks whether all slots of the controller are initialized successfully. If one slot is not initialized successfully, the system cyclically checks the slot initialization status until all slots are initialized successfully .
4. The slot initialization fails. As a result, the interface card repair process is cyclically checked and the repair process cannot be executed. When the version number is checked again, the version verification is inconsistent with the target version. As a result, the interface card upgrade fails.
[Location Method]
1. Confirm the symptom.
After a spare controller is replaced during the upgrade, the T4 interface cards on the spare controller fail to be upgraded. The T4 interface cards include TOE optical interface cards, TOE electrical interface cards, and FCoE cards.
2. Confirm the scenario.
Replace the spare controller during the upgrade.
3. Confirm the version.
See the scope of application.
This problem occurs when the preceding three conditions are met.
Appendix: Log Locating and Analyzing
a. OptiX OSN 6800/6900 V3, V300R003C00 and later versions.
b. Replace the spare controller during the upgrade.
c. After the T4 chip interface card is upgraded, search for [Adapter report Bus(0x8f) need chip recover.][NIC][t4_report_chip_recovery,14128][upgrade_atom] in the log. In the log, (0x8f) indicates the PCIe bus number of the interface card. The interface card may be different in different slots. t4_report_chip_recovery is the name of the function for the card driver to report the chip recover event, and upgrade_atom is the thread for upgrading the module. This log indicates that the interface card of the T4 chip reports the chip recover event during the upgrade.
d. Search for [ERR][The device is not supported by pciehp driver!][PCIE_HP][PCIEHP_AllSlotInitOK,1939][ErrThread_80:3] in the log file. In the log file, 1939 is the log line number and may vary with software versions. ErrThread_80:3 is the name of the thread for processing chip errors. 80:3 varies with devices. This log is repeatedly printed, indicating that some slots are not initialized during the chip error processing. After the slot initialization is complete, the chip error event cannot be processed and the interface card upgrade cannot take effect.
e. During the upgrade, the T4 chip interface card fails to be upgraded. T4 chip interface cards include TOE optical interface cards, TOE electrical interface cards, and FCoE cards.
f. This problem occurs if the preceding five conditions are met.
[Solution]
Temporary solution:
During the upgrade, if an interface card fails to be upgraded for a controller that is replaced with a spare part, restart the controller after the upgrade process stops.
(Do not restart the controller by removing and inserting the controller.
The problem cannot be rectified by restarting the controller.) to retry the upgrade.
