ICMP delay with jumbo frames Highlighted

169 0 6 2

Hello all,

this case mainly talks about the ICMP delay caused by the packet processing mechanism of the CE switch.

Problem Description

The CE12808 switch is connected to the CRS_Router Cisco switch through four Layer 3 interfaces. After the interconnection, it's found that the ICMP delay between the two devices is long.


imgDownload?uuid=bd70c794892a43068942105


To check whether the link is normal, we make an ICMP test with large Packets and see the following on Huawei & Cisco.


1. From Huawei to Cisco :

~Huawei]ping -m 30 -t 8000 -c 100 -s 9152 -a  192.168.23.2 -f  192.168.23.1 

  PING 192.168.23.1: 9152  data bytes, press CTRL_C to break

    Reply from 192.168.23.1: bytes=9152 Sequence=1 ttl=255 time=5 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=2 ttl=255 time=6 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=3 ttl=255 time=3 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=4 ttl=255 time=3 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=9 ttl=255 time=17 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=10 ttl=255 time=59 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=11 ttl=255 time=93 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=12 ttl=255 time=93 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=13 ttl=255 time=93 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=14 ttl=255 time=92 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=15 ttl=255 time=93 ms

                                      ...

    Reply from 192.168.23.1: bytes=9152 Sequence=24 ttl=255 time=92 ms

    Reply from 192.168.23.1: bytes=9152 Sequence=25 ttl=255 time=93 ms


  --- 192.168.23.1 ping statistics ---

    100 packet(s) transmitted

    100 packet(s) received

    0.00% packet loss

    round-trip min/avg/max = 3/84/94 ms


2. From Cisco to Huawei :


Cisco# ping   192.168.24.2 source   192.168.24.1 size 9180 donotfrag count 30000 interval 5


Fri Jul  1 03:50:57.844 GMT


Type escape sequence to abort.


Sending 30000, 9180-byte ICMP Echos to 192.168.24.2, timeout is 2 seconds:


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


!!!!!!!!!!!!!


Success rate is 100 percent (153/153), round-trip min/avg/max = 1/90/94 ms



There seems to be some ISSUE on the link interconnect because the delay should not exceed 15 or 20 ms.

Handling Process

Before checking the issue we need to be clear about MTU for Cisco & Huawei:

A. Cisco Interface MTU is 9194: The ICMP test packet length is 9180 bytes through Cisco CRS Ios XR not consider 14 bytes for Ethernet on ICMP Test.

B. Huawei Interface MTU is 9180: The ICMP test packet length is 9152 bytes through  Huawei does not consider 14 bytes for Ethernet, 8 bytes for ICMP, and 6 bytes for VLAN.

1. At the beginning of the issue, we thought the issue was on Optical Fibers and Transceivers, so our plan was to verify physical links and made physical loops for verifying the Transceiver. After this, the issue continues.

 

[~Huawei]display interface 10GE8/0/46  transceiver  verbose  | in DBm

   Current RX Power (dBm)                :-2.79

   Default RX Power High Threshold (dBm) :2.50

   Default RX Power Low Threshold (dBm)  :-16.40

   Current TX Power (dBm)                :-3.75

   Default TX Power High Threshold (dBm) :2.50

   Default TX Power Low Threshold (dBm)  :-10.20

Cisco Ten 0/4/0/0 <<>> Huawei 10GE5/0/46

Cisco#sh controllers tenGigE 0/4/0/0 phy  | in dBm

Fri Jul  1 03:18:00.095 GMT

                Tx power: -3 dBm (4307 in units of 0.1 uW)

                Rx power: -5 dBm (2597 in units of 0.1 uW)


<Huawei>display  interface 10GE5/0/46  transceiver verbose  | in dBm

   Current RX Power (dBm)                :-3.90

   Default RX Power High Threshold (dBm) :2.50

   Default RX Power Low Threshold (dBm)  :-16.40

   Current TX Power (dBm)                :-3.26

   Default TX Power High Threshold (dBm) :2.50

   Default TX Power Low Threshold (dBm)  :-10.20


2. Verify the packets for ICMP statistics and check if there is some issue with processing :


<Huawei>disp current-configuration configuration include-default | i fast

undo arp fast-reply disable

undo icmp echo-reply fast disable


[~Huawei]display icmp fast-reply statistics slot 5

------------------------ Display ICMP Statistics -------------------------------

Received packets:                                                              

     request packets:                             0                            

     invalid request packets:                     0                            

     failed to get vrf:                           0                            

     destination is not host ip:                  0                            

     failed to get ctrl word:                     0                            

Send packets:                                                                   

     successful reply packets:                    0                            

     failed reply packets:                        0                            


--------------------------------------------------------------------------------


[~Huawei]display icmp fast-reply statistics slot 8

------------------------ Display ICMP Statistics -------------------------------

Received packets:                                                               

     request packets:                             0                            

     invalid request packets:                     0                            

     failed to get vrf:                           0                            

     destination is not host ip:                  0                            

     failed to get ctrl word:                     0                            

Send packets:                                                                  

     successful reply packets:                    0                            

     failed reply packets:                        0                            


--------------------------------------------------------------------------------


[~Huawei]display icmp fast-reply statistics slot 1


------------------------ Display ICMP Statistics -------------------------------

Received packets:                                                              

     request packets:                             0                             

     invalid request packets:                     0                            

     failed to get vrf:                           0                            

     destination is not host ip:                  0                             

     failed to get ctrl word:                     0                            

Send packets:                                                                  

     successful reply packets:                    0                            

     failed reply packets:                        0                            

--------------------------------------------------------------------------------


[~Huawei]display icmp fast-reply statistics slot 4

------------------------ Display ICMP Statistics -------------------------------

Received packets:                                                              

     request packets:                             0                            

     invalid request packets:                     0                            

     failed to get vrf:                           0                            

     destination is not host ip:                  0                            

     failed to get ctrl word:                     0                             

Send packets:                                                                  

     successful reply packets:                    0                            

     failed reply packets:                        0                            

--------------------------------------------------------------------------------

Root Cause

The Ping Packet Processing on a CE Switch is like follows :

imgDownload?uuid=218fdfcb05dc45c28d3e53b

1. The green line in the preceding figure shows ping packet processing. The forwarding plane sends ping packets to the control plane through the CPU. The processing latency mainly includes the latency on the forwarding plane and the latency on the control plane. The latency on the forwarding plane is at the microsecond level and can be ignored. The latency on the control plane is determined by the CPU usage. If the CPU usage is high, the latency increases.

2. The red lines show service data flow processing. Service data packets are directly forwarded by the forwarding plane and are not processed by the control plane. The latency is at the microsecond level.

Analysis of Latency and Jitter During Ping Packet Transmission Between CE and Cisco Switches :

Ping packet processing is as follows:

1.  The packet receiving process on the CE switch is as follows:

Control plane of the Cisco switch -> packet sending by the CPU of the Cisco switch -> forwarding plane of the Cisco switch -> forwarding plane of the CE switch -> packet receiving by the CPU of the CE switch -> control plane of the CE switch

2. The packet forwarding process on the CE switch is as follows:

Control plane of the CE switch -> packet sending by the CPU of the CE switch -> forwarding plane of the CE switch -> forwarding plane of the Cisco switch -> packet receiving by the CPU of the Cisco switch -> control plane of the Cisco switch.

Solution

The latency of ping packets mainly includes the processing latency on the Cisco switch and the processing latency on the CE switch. The processing latency on the CE switch mainly includes the latency on the forwarding plane and the latency on the control plane.

The latency on the forwarding plane is at the microsecond level and can be ignored. The latency on the control plane is determined by the CPU usage. If the CPU usage is high, the latency increases.

Because of the CPU performance and software calculation priority of the CE switch, a little latency and jitter may occur for ping packets that need to be processed by the CPU. However, data packet forwarding is not affected on the CE switch because data packets are forwarded by the hardware and software processing is not involved.

Suggestions and Summary

As we mentioned this issue is just related to IMCP Test, not with Traffic of Services.

That's true because when we put traffic on the Internet there is no issue with users.

Before the ICMP test, ensure that the test does not affect service traffic processing.


That's all, please feel free to ask me any questions!

  • x
  • convention:

Comment

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login