Keywords :
Qos port-queue command on ATN950B, POS Link Migration to GE
Abstract :
In a migration activity to move from POS links to GE in a customer network, 2G service went down due to inadequate bandwidth reservation for the EF class for the 2G service which was unintentional. Metric for command for bandwidth shaping on ATN is in Kbits while same command on ATN has metric in Mbits.
Fault Type :
Operation and maintenance>>QoS setting
Issue Description :
Fault Symptom :
After a migration activity from POS to GE, QoS configuraiton on the target topology caused 2G service to be done. 2G voice calls were not successful
Version Information :
- NE40E-8 - V600R001C00SPC800
- NE40E-X3 - V600R003C00SPC00
- ATN950B - V200R003C00
Network Topology before Migration :
Network Topology After Migration :
- POS link from NE40E-8-SiteB to NE40E-X3-SiteA-02 migrated to GE.
- POS Link from NE40E-8-SiteC to NE40E-8-SiteD decommissioned as it had low capacity (155Mbps)
- Since POS link from NE40E-8-SiteC to NE40E-8-SiteD is decommissioned, backup link indicated in yellow now starts carrying traffic as main Tunnel is down.
- ATN950B SiteA à NE40E-8-SiteC à NE40E-8-SiteB --> NE40E-X3-SiteA-02 à NE40E-8-SiteA-01 is then defined as the new backup path.
Troubleshooting/Handling Process :
1. The network Operations Center (NOC) was contacted to check the status of 2G sites on the whole metro. Feedback showed there was no physical issue for all sites but they noticed some sites could not traffic for 2G.
2. The list of sites that could not traffic were collected to confirm what they all had in common. The common point was these sites are all using ATN950B-SiteA and NE40E-8-SiteC as their Gateway for 2G service. All the other sites with a different GW had their sites traffic on 2G.
3. An FME was sent on site using ATN950B-SiteA as Gateway for 2G service and was requested to try to initiate a call which ended with no success. A call from a working site to the impacted site (where the FME was located) had no success.
4. The status of the Main and backup tunnels of ATN950B-SiteA and NE40E-8-SiteC were checked using an lsp ping test and the tunnels were all confirmed up with the Route Reflector. During all these checks, those same sites could not carry traffic.
5. An attempt was made to block the primary tunnel going through ATN950B-SiteD and we noticed the sites started carrying traffic and 2G calls were going through. At this point the backup path ATN950B SiteA à NE40E-8-SiteC à NE40E-8-SiteB à NE40E-X3-SiteA-02 à NE40E-8-SiteA-01.
6. As 2G service was now successful, we did a node per node configuration check. At this point could not find any issue with the configuration.
7. We now advised to go back to the previous scenario (where 2G voice service was impacted) where the backup path was not used and the 2G service was unsuccessful to do some displays on statistics to see if any issue could be noticed.
a. The following command was run on all point to point interfaces of the following equipment : ATN950B SiteA à NE40E-8-SiteC à NE40E-8-SiteB à ATN950B-SiteD à NE40E-8-SiteA-01 and it was noticed for ATN950B-SiteD, we had discards on the interface Gi0/2/1 in the outbound direction
display port-queue statistics interface gigabitethernet 0/2/1 ef outbound
b. The configuration on the interface was checked and we noticed the following : Compared to the other routers, there was no shaping provided on the P2P interface but for this interface (an old configuration existing on the router ATN950B-SiteD interface)
interface GigabitEthernet0/2/1
description G0/2/1_To_NE40E-8-SiteA-01_Gi4/1/0
…
port-queue ef pq shaping 600 outbound
c. A further check was done to understand the syntax of the command and it was noticed the metric used is the Kbps and not the Mbps as for the case of the NE40E. This means when the 2G traffic reached the interface of this router, it had a reservation of only 0.6Mbps. This caused a high saturation and all packets were dropped. The intention was to have 600Mbps reserved for the EF class. The command was then adjusted as below and the service came back to normal
interface GigabitEthernet0/2/1
description G0/2/1_To_NE40E-8-SiteA-01_Gi4/1/0
…
port-queue ef pq shaping 600000 outbound
Root Cause :
The root cause for the packets being dropped was the inadequate bandwidth reserved on the EF class for the interface on the ATN950B-SiteD. The 0.6Mbps was not enough to carry the 2G traffic (high congestion) so 2G calls could not go through.
Solution :
Correct the shaping value of the queue from 600Kbps to 600000Kbps
Suggestions :
1. It will be good for VRP developers to align the metric value for all routing platforms to be the same to avoid confusion when a particular platform is used.
2. All Engineers working on plaforms should always use the “question mark” to understand the syntax of commands before applying them
Extract / Hedex documentation / Port-queue command / ATN :
Extract / Hedex documentation / Port-queue command / NE40E :
