Hello, everyone.
This post will analyze the ECC storm phenomenon, its causes and its solutions. Then, based on the characteristics of the ECC protocol, it will provide reasonable suggestions from the perspective of network planning in order to reduce the probability of ECC storm problems and improve the efficiency of solving such problems.
Problem Description
Symptom 1: A large number of NEs go offline.
After an ECC storm occurs, a large number of NEs go offline. In most cases, all NEs except the gateway NE cannot be logged in. After querying the routing table, it was found that some random routes existed and the distance of these routes was large.
Symptom 2: Network flapping.
The ECC network flaps and a large number of ECC route update packets are continuously transmitted on the network. As a result, the ECC routes of some NEs change continuously and the ECC communication is intermittent and unstable.
ECC storms occur on both SDH and WDM networks. This is because they all use the HWECC protocol to transmit network management information.
Cause analysis
ECC storms are caused by the following reasons:
1. The ECC network scale is too large
Like IP RIP, HWECC is a distance-vector protocol and cannot prevent route loops. Bulk routing information must be broadcast after a route change. Therefore, HWECC is designed as an IGP routing protocol for a small network. In the case of RIP, an ECC spans up to 16 hops. In the case of HWECC, however, an ECC travels up to 64 hops by default. In the case of a route change, the cyclic invalid data information must run through 64 hops before being discarded. For a large network, this is no doubt a disastrous load to the network bandwidth. Few bandwidth resources are available for transferring ordinary routing data. In addition, the other NEs must use some bandwidth resources to set up new MAC connections and routes. Thus, a vicious circle forms. For a long time, the routes cannot be converged. An ECC storm finally occurs. This frequently happens when a fiber cut occurs, or when the main control board or line board is being replaced.
2. In the complex or ring network, ECC storms may occur when fibers are severely degraded
When the bit errors of the optical line are large or the packet loss is severe, the ECC protocol generates some disordered or outdated routes when broadcasting packets. If the network is a ring network, these routes are cyclically sent to the other direction, which is difficult to disappear within a period of time. Therefore, we see many routes with very large distances. The more bit errors, the more disordered routes are generated. The ring network is the amplifier of these disordered routes. As a result, routes of NEs are frequently switched. The priority of the ECC route task is higher than that of the login task. When routes on an NE are frequently switched, the CPU usage of the routing task is high, causing the login task with a low priority not to run for a long time. As a result, the NE is unreachable to the NMS.
3. NE ID conflict
ECC communication between NEs is implemented through ID address identification. Therefore, each NE must have an independent identifier ID. If the IDs are duplicated, ECC route calculation fails. The NMS cannot find the destination when delivering connection requests. As a result, a large number of connection request packets are generated on the network. Then an ECC storm occurs.
4. Communication with a large amount of data
When the configuration data of multiple NEs is uploaded or the version of multiple NEs is upgraded at the same time, a large amount of data is transmitted on the network, causing network congestion. In this case, ECC storms may occur.
In the preceding four cases, there is a high probability that ECC storms are caused in the first and second cases. The probability of ECC storms caused by the third and fourth cases is relatively low.
Solution
The solution consists of two steps:
Step 1: Restore the NMS to manage NEs.
Method 1: Set the maximum distance of ECC.
The maximum distance of ECC is 64 by default. In the actual network, such a large distance is not required. In addition, this maximum distance affects the search scope of ECC route.
Setting the maximum distance of ECC can narrow the range of refreshing the ECC route on the network, thus reducing the probability of an ECC storm. When an ECC storm occurs, set the maximum distance of ECC to 5. Then, after the network becomes stable, increase the maximum distance to stabilize the network gradually.
Run the commands to set the route distance:cm-set-maxdist
For example:
:cm-set-maxdist:5 # The maximum route distance cannot be set to 0.
Method 2: Disable the ECC links around the backbone node.
When disabling the ECC links, you must be familiar with the fiber connections on the network. First, disable the loops at the access layer and then isolate certain devices from the current ECC network. After the ECC is stable, gradually release the devices.
When disabling the remote optical ports, do not disable the route to the NMS. Make sure that the disabled ECC can be accessed on the NMS.
Step 2: Eliminate the root cause of the fault.
1. Troubleshooting bit errors on the line.
Locate the optical port with a large number of bit errors, disable the DCC channel, check the fiber, or replace the optical board.
2. Check for NE ID conflicts.
Change a duplicate NE ID to ensure that the IDs of all NEs managed by the same NMS are unique. If the NMS manages NEs of the Transmission domain, access domain, and IP domain at the same time, the NE IDs of these NEs also must be unique.
3. Suspend the operations that require large amounts of data to be transferred across the DCC network.
Preventive measures
In actual application scenarios, there is no way to prevent bit errors from occurring on fibers or ring networks. However, the following precautions can be taken to reduce the probability of ECC storms and improve the efficiency of handling ECC storms:
1. Proper subnetting
Properly control the ECC network scale. It is recommended that each subnet contain no more than 64 NEs.
2. Do not use automatic extended ECC.
When three or more NEs are connected at a site, automatic ECC extension is not used. If devices are connected only through network cables, manual ECC extension is recommended. This is because the automatically extended ECC forms a very complex ring network.
3. Properly arrange the gateway location.
When an ECC storm occurs, only the gateway can be logged in. Therefore, the location of the gateway on the network is very important for fault prevention and recovery. When an ECC storm occurs, you can set the DCC on the gateway or the maximum route distance to cut the loop. This way, disordered routes caused by bit errors will not flap on the loop. Therefore, on a ring network, ensure that each ring has a GNE. For example, in the following network, NE (9-2335) is the most suitable gateway while NE (9-1213) is the least suitable gateway.
4. Properly plan the network to avoid duplicate NE IDs.
Thanks!
