S Series switches packet loss troubleshooting【All About Switches】 Highlighted

Latest reply: Aug 6, 2017 10:46:03 3771 3 1 2

This post refers to the S Series switches packet loss troubleshooting as part of the All About Switches section. 


Contents

1 About This Document 1

[url=http://support.huawei.com/huaweiconnect/enterprise/]2 Packet Loss Problem Locating and Handling. 2


2.1 Checking Whether Packets Have BeenLost 5

2.2 Checking the PC Where Packet LossOccurs. 5

2.3 Checking Whether the Physical Statusof Interfaces Is Down. 5

2.4 Checking CRC Errors in InboundDirection of Interfaces. 8

2.5 Checking the Discard Count in theOutbound Direction. 9

2.6 Checking Loops. 10

2.7 Checking Attacks. 14

2.8 Checking Whether the Rate of PacketsSent to the CPU Exceeds the Limit 17

2.9 Checking the Related Configurations. 18

2.10 Determining Packet Loss LocationBased on Traffic Statistics. 23

2.10.1 Loading the Latest Patch Matching the System SoftwareVersion. 25

2.10.2 Resetting or Replacing the Problematic Cards. 26

2.10.3 Checking the Physical Links. 26

2.11 Contacting Technical SupportPersonnel 28

3 Typical Cases. 29

3.1 A Few Packets Are Dropped 2 MinutesAfter Multicast Service Is Enabled. 29

3.2 Management User Cannot Telnet to aSwitch. 31

3.3 Ping Packets from Monitoring Server toServers Are Dropped. 32

3.4 Inbound Packets Are Dropped. 33

3.5 Pixelation Appears on the Screen Whena Switch Is Connected to Two Receivers. 34

3.6 User Hosts Go Online Slowly andPinging the Gateway Has a Delay. 36

3.7 Packets Are Dropped When a Switch IsConnected to a Third-Party Device. 40

3.8 Packets Are Lost When Access UsersPing a Stacked Switch. 41

3.9 Users Connected to a Switch CannotAccess the Network. 42

3.10 Packets Are Lost When Users Access aServer 43

3.11 Artifacts Occur When Users WatchVideos. 43

3.12 Pixelization Occurs When Users WatchVideos. 45

3.13 Pixelization Occurs When Users WatchIPTV During Peak Hours. 47

3.14 Image Pauses Occur When Users WatchVideos. 49

4 Packet Loss Prevention Measures. 51

5 Appendix. 53

5.1 Configuring Traffic StatisticsCollection. 53




1 About This Document


Overview
When managing and maintaining a network, wemay encounter intermittent network disconnection caused by transmission delayor slow network access speed. Most of these symptoms are caused by packet loss.
Quickly isolating and rectifying thefailures are important.
This document analyzes the packet losssymptoms on the network, describes the fault locating and fixing methods, andprovides typical cases and reference for you to fix the packet loss problems.

The functions and commands supported on aswitch may vary depending on the product model and software version. This documentuses V200R008C00 as an example. For the functions and commands used on yourswitch, see the documentation of the specific version.
ChangeHistory

   Issue
   
   Release Date
   
   Description
   
  02
  
  2017-01-31
  
  Second commercial release.
  
  01
  
  2016-12-30
  
  Initial official release
  



2 Packet Loss Problem Locating and Handling


When a packet loss occurs, determine thelocation where packets are dropped, analyze the cause of the packet loss, andthen rectify the fault accordingly. Figure 2-1 showsthe fault locating process.
Figure 2-1 Packet loss problem locating and handling

 
This document uses a campus network as anexample to describe the packet loss problem locating and handling methods.
Figure2-2 showsthe diagram of the campus network. Users A, B, and C are connected to accessSwitch_3 and Switch_2. User D and user E are connected to access Switch_4.Switch_3, Switch_2, and Switch_4 are connected to the core Switch_1. Switch_1is connected to the Internet through the firewall.
Figure 2-2 Campus network

 
User A complains that network access speedis slow and web pages cannot be opened sometimes. Other users do not complainabout this. When user A's PC pings a public network address, the ping packet isdropped.
2.1  Checking Whether Packets Have Been Lost
2.2  Checking the PC Where Packet Loss Occurs
2.3  Checking Whether the Physical Status ofInterfaces Is Down
2.4  Checking CRC Errors in Inbound Direction ofInterfaces
2.5  Checking the Discard Count in the OutboundDirection
2.6  Checking Loops
2.7  Checking Attacks
2.8  Checking Whether the Rate of Packets Sent tothe CPU Exceeds the Limit
2.9  Checking the Related Configurations
2.10  Determining Packet Loss Location Based onTraffic Statistics
2.11  Contacting Technical Support Personnel
2.1 Checking Whether Packets Have Been Lost

The symptoms of packet loss are as follows:
l  When users go online:
          Connection speed is unstable.Web pages are opened at a low speed, or some parts of web pages or the entireweb pages cannot be displayed.
          Video is frozen or artifactsappear on the screen.
          The instance messaging tool,such as QQ, is frequently disconnected or login times out.
          File downloading is slow.
l  When a switch is working:
          The ping operation on theswitch times out.
          A port cannot forward data.
          The login of an administratortimes out.
          Service is frequentlyinterrupted.
If one or more symptoms appear, packet losshas occurred.
2.2 Checking the PC Where Packet Loss Occurs

Check the PC where packet loss occurs.
For example, check whether the networkcards work normally and whether the cable between PC and connected device isnormal. Solution: Disconnect the PC from the network and scan virus on the PC,check the cable, reinstall the operating system, and check network cards.
If the PC is normal but the fault persists,go to the next step.
2.3 Checking Whether the Physical Status of Interfaces Is Down

If the physical status is Down, or theduplex mode or negotiated rate of the local interface is different from that onthe remote end, the interface status is abnormal.
1.        Run the display interface interface-typeinterface-number command to check the interface running status.
For example, the configuration ofGE1/0/2 on Switch_3 is as follows:
<HUAWEI> display interfacegigabitethernet 1/0/2 
GigabitEthernet1/0/2 current state : DOWN //Current physical status of the interface
Line protocol current state : DOWN 
Description: 
Switch Port, Link-type : access(negotiated), 
PVID : 1, TPID : 8100(Hex), The Maximum Frame Length is 9216 
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is000b-0974-a475 
Last physical up time : 2016-08-10 13:09 Last physical down time :2016-08-10 13:09 
Current system time: 2016-08-10 13:09 
Port Mode: COMMON FIBER  //Interfaceworking mode. COMMON COPPER indicates an electrical interface, and COMMON FIBERindicates an optical interface.
Speed : 1000, Loopback: NONE  //Therates, loopback interface, and link status on two ends must be consistent.
Duplex: FULL, Negotiation: ENABLE  //Theduplex mode, negotiation status, and link status on two ends must beconsistent.
---- More ----
          If the command output includes"current state : UP", the interface works normally. Then skip thissection.
          If the command output includes"current state : Administratively down", the interface has been shutdown.
Run the interface interface-typeinterface-number command in the system view to enter the interface view,and then run the display thiscommand to check the interface configuration. If the interface has been shutdown using the shutdown command, runthe undo shutdown command in theinterface view.
          If the command output includes"current state : DOWN", check whether the negotiation status, rates,duplex modes, and cable modes on two ends are consistent.
Run the display interface command on two ends of the link. The informationshown in Table 2-1 isdisplayed.
Table 2-1 Checking the duplex modes, rates, and negotiation modes on two ends

   Item
   
   Description
   
   Follow-up Procedure
   
  Negotiation
  
  Negotiation status:
  l   ENABLE: auto-negotiation
  l   DISABLE: non-auto-negotiation
  
  The two interfaces must work in the same  negotiation mode (both in auto-negotiation or non-auto-negotiation mode).
  1.     Run the negotiation auto command in the interface view to enable  auto-negotiation.
  2.     If the fault persists,  disable auto negotiation and forcibly set the same rate and duplex mode on  the interfaces.
  
  Speed
  
  Current rate
  
  l   If the interface rates are  inconsistent in the auto-negotiation mode, run the restart command to restart the interface, making them consistent.
  l   If the interface rates are  inconsistent in the non-auto-negotiation mode, run the speed { 10 | 100 | 1000 } command in the interface view, making them consistent.
  NOTE
  If an electrical interface works  properly at 10 Mbit/s or 100 Mbit/s but does not work properly at 1000  Mbit/s, check whether the cable works properly. If not, replace the cable.
  
  Duplex
  
  Duplex mode of the interface
  l   FULL: full duplex
  l   HALF: half duplex
  
  l   If the duplex modes on two  ends are inconsistent in the auto-negotiation mode, run the restart command to restart the  interface, so that they can negotiate the same mode. Alternatively, run the auto duplex  full  command in the interface view to enable full duplex mode.
  l   If the duplex modes of the  interfaces are different in non-auto-negotiation mode, run the duplex full command in the interface  view to enable full duplex mode.
  
  Mdi
  
  Ethernet cable type supported by the  interface:
  l   across: crossover cable
  l   auto: cable type  automatically identified That is, the interface can connect to a  straight-through cable or a crossover cable.
  l   normal: straight-through  cable
  
  Ensure that the cable modes and types on  two ends are consistent.
  By default, the cable mode is auto. If  the cable mode is not auto, run the mdi  auto command in the interface view to change it to auto.
  

 
          If the command output includes"current state : ERROR DOWN (down-cause)", the interface is shut downdue to an error. Find out the cause based on the down-cause field.
For details, see the troubleshootingsession about physical status Down of Ethernet interfaces.
If the interface status is still Down,go to the next step.
2.        Connect the cable to anotheridle interface, and check whether the interface status is normal.
          If the physical status is stillDown, contact technical support personnel.
          If both interfaces are Up, workin full duplex mode, and are enabled with auto-negotiation, but packet losspersists, see the next section.
2.4 Checking CRC Errors in Inbound Direction of Interfaces


This section is separated from"Checking Interface Status" because packet loss caused by congestionoften happens on live networks.
Check whether cyclic redundancy check (CRC)errors occur on the physical interfaces along the transmission path of packets,and whether the number of CRC errors increases continuously.
If the number of CRC errors is not 0 andkeeps increasing, the interface is receiving CRC error packets. The errorpackets are generated due to the incorrect physical link status or deviceproblems.
Rectify the link problem according to 2.10.3 Checking the Physical Links. Ifthe problem persists, contact technical support engineers.
<HUAWEI> display interface gigabitethernet 1/0/2
GigabitEthernet1/0/2 current state : UP
Line protocol current state : UP
Description:
Switch Port,PVID :    1,TPID :8100(Hex),The Maximum Frame Length is 1600
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 0025-9e80-2494
Current system time: 2016-08-10 13:09-05:13
Port Mode: COMMON COPPER
Speed : 1000,  Loopback: NONE
Duplex: FULL,  Negotiation: ENABLE
Mdi   : AUTO
Last 300 seconds input rate 760 bits/sec, 0 packets/sec
Last 300 seconds output rate 896 bits/sec, 0 packets/sec
Input peak rate 12304 bits/sec,Record time: 2016-08-10 13:09
Output peak rate 14568 bits/sec,Record time: 2016-08-10 13:09
Input:  28643 packets, 2734204 bytes
Unicast        :               20923,Multicast          :                7703
Broadcast      :                  17,Jumbo              :                   0
CRC            :                   0,Giants             :                   0
Jabbers        :                   0,Fragments          :                   0
Runts          :                   0,DropEvents         :                   0
Alignments     :                   0,Symbols            :                   0
Ignoreds       :                   0,Frames             :                   0
Discard        :                 474,Total Error        :                   0
Output:  68604 packets, 8057155 bytes
Unicast        :               20429,Multicast          :               14054
Broadcast      :               34121,Jumbo              :                   0
Collisions     :                   0,Deferreds          :                   0
Late Collisions:                  0,ExcessiveCollisions:                  0
Buffers Purged :                   0
Discard        :                   0,Total Error        :                   0
Input bandwidth utilization threshold : 100.00%
Output bandwidth utilization threshold: 100.00%
Input bandwidth utilization  : 0.01%
Output bandwidth utilization : 0.00%
2.5 Checking the Discard Count in the Outbound Direction


This section is separated from "CheckingInterface Status" because packet loss caused by congestion often happenson live networks.
When network resource is insufficient andtransmission rate is lowered, delay occurs. If a network has a burst ofmulticast traffic or run multiple services concurrently, congestion may occur.The traffic burst occupies excessive bandwidth, some packets are dropped.
1.        Check whether there arediscarded packets on the interface.
Run the display interface interface-typeinterface-number command in any viewor the display this interfacecommand in the interface view to view outbound packet statistics on theuser-side interface. If the Discard count is not 0, the port has beencongested. Check whether the number of discarded packets increases.
          If not, the problem is irrelevantto packet loss. Then skip this section.
          If so, the problem is caused bypacket loss. Then go to the next step.
[HUAWEI]display interfaceGigabitEthernet 1/0/2
GigabitEthernet1/0/2 current state : UP
Line protocol current state : UP
Description:link-to-OLT-LB
Switch Port, PVID :1, TPID : 8100(Hex), The Maximum Frame Length is 1600
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is4c1f-cc45-b1c0 
Current system time: 2016-08-10 13:09
Port Mode: COMMON COPPER
Speed : 1000,Loopback: NONE
Duplex: FULL,Negotiation: ENABLE
Mdi: AUTO,Flow-control: DISABLE
Last 300 seconds input rate 99894144 bits/sec, 141895 packets/sec
Last 300 seconds output rate 190939848 bits/sec, 271220 packets/sec
Input peak rate 173002368 bits/sec, Record time: 2016-08-10 13:09
Output peak rate 346005880 bits/sec, Record time: 2016-08-10 13:09
Input:175946456 packets, 15483288128 bytes
Unicast:0,Multicast:0
Broadcast:175946456,Jumbo:0
Discard:0, Pause:0
Total Error:0
CRC:0,Giants:0
Jabbers:0,Fragments:0
Runts:0,DropEvents:0
Alignments:0,Symbols:0
Ignoreds:0,Frames:0
Output:348119287 packets, 30634557621 bytes
Unicast:0,Multicast:773
Broadcast:348118514,Jumbo:0
Discard:3769937,Pause:0
Total Error:0
Collisions:0,ExcessiveCollisions:0
Late Collisions:0,Deferreds:0
Buffers Purged:0
2.        Set the buffer mode to enhancedand check whether the number of discarded packets increases.
Generally, the buffer size on aninterface is small. When the traffic rate on an interface reaches 50% to 60% ofthe interface bandwidth, packets are lost on the interface. After the buffermode on the interface is set to enhanced, an interface can preempt more dynamicbuffer and the interface has a higher capability to cope with traffic burst.This can relieve packet loss caused by congestion.
<HUAWEI> system-view
[HUAWEI] interface gigabitethernet 1/0/2 
[HUAWEI-GigabitEthernet1/0/1] qos burst-mode enhanced

The X1E series LPUs do not support thiscommand.
When the buffer mode is set to enhanced,the qos burst-mode (interface view)command and the qos burst-mode(system view) command cannot be used together. In addition, the precedingcommands cannot be used together with qosqueue length.
Check the Discard count on the interfaceagain.
          If the number of discarded packetsdoes not increase, congestion is eliminated. Check whether packet loss isfixed. If not, skip this section and go to the next section.
          If the number of discardedpackets increases or the device does not support the qos burst-mode command, optimize the network.
3.        Optimize the network.
Consider the following methods:
          Configure traffic rate limitingor shaping in the inbound direction.
Traffic burst is the major reason forunexpected packets loss. When the burst packet size exceeds the interfacebuffer, service packets are lost, affecting user services. Configure trafficrate limiting or shaping on the upstream device can relieve traffic burst orreduce burst packet size, and the possibility that congestion occurs ondownstream devices is reduced.
          Provide differentiated serviceson ports. The key services are placed in high-priority queues, which can beprocessed first upon congestion.
If an interface runs many services,including high-priority services (such as voice and video) and low-priorityservices (such as network access). You can specify different priorities forhigh-priority services on the upstream device or configure priority mapping inthe ingress of a device, to ensure that key services enter the high-priorityqueues. Configuring PQ scheduling in the egress can ensure that high-priorityservices are scheduled first.
          If multiple flows conflict,increase the link bandwidth between devices or add more interfaces to theEth-Trunk for load balancing.
          If multicast service isconfigured, adjust the packet sending mode on the multicast source server torelieve the congestion.
2.6 Checking Loops

Loop is a common cause for packet loss, andis difficult to detect loops. For example, on a large-sized network, if theadministrator incorrectly connects switch interfaces to cause a loop, packetswill be dropped.
1.        Check whether the followingsymptoms occur:
In addition to packet loss, loops maycause the following symptoms:
          Run the display interface brief | include up command to view trafficstatistics on the Up interface. If the interface has a loop, the InUti andOutUti values increase, or even approximate 100%. The values outdistanceservice traffic volume.
First query:
<SwitchA> display interface brief | include up 
 ... 
Interface                   PHY   Protocol  InUti OutUti  inErrors  outErrors                                                       
GigabitEthernet0/0/2        up    up       0.56%  0.56%          0          0                                                       
 ...    
Last query:
<SwitchA> display interface brief | include up 
...
Interface                   PHY   Protocol InUti OutUti   inErrors  outErrors                                                      
GigabitEthernet0/0/2        up    up         76%    76%          0          0                                                      
...
          The display interface command output shows a large number of broadcastpackets received on an interface.
          Indicators of interfaces in theVLAN where a loop has occurred blink faster than usual.
          The CPU usage of a switchexceeds 80%.
<SwitchB> display cpu-usage 
CPU Usage Stat. Cycle: 60 (Second)  
CPU Usage            : 95% Max: 97%   
CPU Usage Stat. Time : 2013-08-21 16:38:44  
CPU utilization for five seconds: 95%: one minute: 95%: five minutes: 95%  
Max CPU Usage Stat. Time : 2016-08-10 13:09.  
 ....
Run the display cpu-usage command to check CPU usage. Network loops maylead to continuous high CPU usage, and the switch may fail to process packetsand discards them.
          MAC flapping frequently occurs.
n   Run the display trapbuffercommand to check whether a MAC flapping alarm is reported,
The MAC flapping alarm is asfollows:
L2IFPPI/4/MFLPVLANALARM:OID1.3.6.1.4.1.2011.5.25.160.3.7 MAC move detected, VlanId = 22, MacAddress =0000-5e00-0116, Original-Port = Eth-Trunk1, Flapping port = Eth-Trunk11. Pleasecheck the network accessed to flapping port.
Check whether the MAC flappingoccurs on the device with ping packet lost.
n   Run the mac-address flappingdetection command to configure MAC address flapping detection, and then runthe display mac-address flapping recordcommand to check whether MAC address flapping is occurring.
<HUAWEI> system-view
[HUAWEI] mac-address flapping detection
<HUAWEI> display mac-addressflapping record
 S : start time                                                                
 E : end time                                                                  
(Q) : quit VLAN 
(D) : error down 
------------------------------------------------------------------------------
Move-Time             VLAN  MAC-Address  Original-Port Move-Ports   MoveNum
-------------------------------------------------------------------------------
S:2016-08-10 13:09 300  0000-0000-0007Eth-Trunk1   Eth-Trunk2   81
E:2016-08-10 13:09
 
-------------------------------------------------------------------------------
Total items on slot 2: 1
n   Run the display mac-addresscommand multiple times. If the MAC address is learned by different interfaces,MAC flapping exists.
<HUAWEI> display mac-address
------------------------------------------------------------------------------- 
MAC Address          VLAN/VSI                    Learned-From        Type       
-------------------------------------------------------------------------------
0022-0022-0033       100/-                       GE1/0/1             dynamic 
0000-0000-0001       -/HUAWEI                    GE1/0/2             static 
-------------------------------------------------------------------------------
Total items displayed = 2
          Loop alarms are generated afterloop detection is enabled.
Table2-2 liststhe loop alarms.
Table 2-2 Loop alarms on the switch

   Alarm ID
   
   Alarm Message
   
   Description
   
  1.3.6.1.4.1.2011.5.25.174.3.1
  
  LDT/4/DetectLoop: OID [oid] The port  detected loop. (InterfaceIndex: [INTEGER] InterfaceName: [OCTET] VlanListLow:  [OCTET] VlanListHigh: [OCTET])
  
  If the packets that a port sends are  returned to the port through the local VLAN, it indicates the packets are  looped. A loop may cause a broadcast storm.
  This alarm is generated when a loop is  detected.
  
  1.3.6.1.4.1.2011.5.25.174.3.2
  
  LDT/4/LoopResume: OID [oid] The detected  loop is removed. (InterfaceIndex: [INTEGER] InterfaceName: [OCTET]  VlanListLow: [OCTET] VlanListHigh: [OCTET])
  
  This recovery notification is generated  when the packet loop of the port is cleared.
  
  1.3.6.1.4.1.2011.5.25.174.3.3
  
  LBDT/4/PORTTRAP: OID [OID] Loopback  exists on interface([INTEGER1]) [OCTET1] ([OCTET2]), loopback detection  status: [INTEGER2].(1:normal; 2:block; 3:shutdown; 4:trap; 5:nolearn;  6:quitvlan)
  
  A loop is detected on the Layer 2 network  connected to the interface.
  
  1.3.6.1.4.1.2011.5.25.174.3.4
  
  LBDT/4/PORTTRAP: OID [OID] Loopback is  removed on interface([INTEGER1]) [OCTET], loopback detection status:  [INTEGER2].(1:normal; 2:block; 3:shutdown; 4:trap; 5:nolearn; 6:quitvlan)
  
  The loop on the interface is removed.
  

 
Select an appropriate method based on theloop information and networking.
1.        Observe interface indicatorsand collect traffic statistics on interfaces to locate the interfacesundergoing broadcast storms.
2.        Check the devices hop by hopaccording to the topology to locate the devices that cause the loop.
3.        Locate the interface thatcauses the loop and shut down the interface to remove the loop.
GE0/0/2 may have a loop, so ping packetson switch B are dropped. To determine whether the packet loss on switch B iscaused by loop, shut down GE0/0/2 of switch B, and then perform a ping test.
# Shut down GE0/0/2 of switch B.
<SwitchB> system-view 
[SwitchB] interface gigabitethernet0/0/3 
[SwitchB-GigabitEthernet0/0/3] shutdown 
[SwitchB-GigabitEthernet0/0/3] quit
# Perform a ping test.
<SwitchB> ping -c 100 192.168.2.21 
 PING 192.168.2.21: 56  data bytes, press CTRL_C to break      
    Reply from 192.168.2.21: bytes=56Sequence=1 ttl=255 time=1 ms    
  ... 
    Reply from 192.168.2.21: bytes=56Sequence=100 ttl=255 time=2 ms                                                                  
                                                                                                                                      
  --- 192.168.2.21 ping statistics---                                                                                              
    100 packet(s) transmitted                                                                                                         
    100 packet(s) received                                                                                                            
    0.00% packet loss                                                                                                                 
    round-trip min/avg/max = 1/1/19ms 
Disabling the interface is not the finalsolution to the loss of ping packets. To solve the problem, eliminate the loopon the connected network. Configure the RRPP, SEP, Smart Link, or STP/RSTP/MSTPprotocol to prevent loops.
4.        if the fault persists after thepreceding operations are performed, collect network information (includinginterface connections) and logs (the log.logfile or the display logbuffercommand output), and provide collected information to Huawei switchagentsHuawei technical support engineers.

This chapter describes only the method oflocating network loops and handling suggestions. For more information, see theS switch loop troubleshooting guide.
2.7 Checking Attacks

When undergoing an attack, a switch is busyprocessing the requests from the attack source, and other service packets aredropped.
Common network attacks, such as ARP, ARPMiss, and DHCP attacks, can cause a high CPU usage on a switch. These attacksare all initiated by sending a large number of protocol packets; therefore,packet statistics on the switch show a large number of packets sent to the CPU.
l  ARP and ARP-Miss attacks
l  DHCP protocol packet attack
l  Other attacks
          ICMP attack
          DDoS attack
          Broadcast attack
          TTL-expired attack
          Sending IP packets of which thedestination address is the target device's IP address
          SSH/FTP/Telnet attack
1.        Run the display cpu-defend statistics command to view statistics about thepackets sent to the CPU, determining whether too many protocol packets arediscarded due to timeout.
a.        Run the reset cpu-defend statistics command to clear statistics about thepackets sent to the CPU.
b.        After several seconds, run the display cpu-defend statistics commandto view statistics about the packets sent to the CPU.
If there are too many packets of aprotocol, determine whether it is normal depending on the networking. If not,there is a high probability that the switch is undergoing a protocol packetattack.
<HUAWEI> reset cpu-defend statistics
<HUAWEI> display cpu-defendstatistics all
Statistics on slot 2:
-----------------------------------------------------------------------------------------------------------
Packet Type         Pass(Bytes)  Drop(Bytes)  Pass(Packets)   Drop(Packets)
-----------------------------------------------------------------------------------------------------------
arp-miss            0           0            0             0
arp-request          40800       35768        600           52600
bgp                0           0            0             0
...
-----------------------------------------------------------------------------------------------------------
The preceding information shows thatthe switch has discarded many ARP request packets. If these packets areabnormal, the switch undergoes an ARP attack.
2.        Configure the attack sourcetracing function to find out the attack source.
If a CPU is busy with many valid orattack packets, services may be interrupted. The switch provides the localattack defense function to protect the CPU. Local attack defense policiesinclude attack source tracing, port attack defense, CPCAR, and blacklist.
For details about local attack defense,see S switch high CPU usage troubleshooting.
a.        Create the local attack defensepolicy based on attack source tracing.
i.         Create an ACL and add thegateway IP address to the whitelist of attack source tracing.
<HUAWEI> system-view
[HUAWEI] acl number 2000 
[HUAWEI-acl-basic-2000] rule 5 permitsource 10.1.1.1 0  //10.1.1.1 is thegateway IP address.
[HUAWEI-acl-basic-2000] quit
ii.       Create the local attack defensepolicy based on attack source tracing.
[HUAWEI] cpu-defend policy policy1
[HUAWEI-cpu-defend-policy-policy1] auto-defendenable  //Enable attack sourcetracing. By default, this function is disabled.
[HUAWEI-cpu-defend-policy-policy1] undoauto-defend trace-type source-portvlan //Set the attack tracing mode to MAC + IP based. By default, attacksource tracing is based on source MAC address, source IP address, sourceinterface, and VLAN ID. To delete unneeded mode, run the undo auto-defend trace-type command.
[HUAWEI-cpu-defend-policy-policy1] undoauto-defend protocol 8021x dhcp icmp igmp tcp telnet ttl-expired udp  //Delete a packet type from attack sourcetracing. By default, the packet types for attack source tracing are 802.1x,ARP, DHCP, ICMP, IGMP, TCP, Telnet, TTL-expired, and UDP.
[HUAWEI-cpu-defend-policy-policy1] auto-defendwhitelist 1 acl 2000  //Add thegateway IP address to whitelist.
[HUAWEI-cpu-defend-policy-policy1] quit
b.        Apply the local attack defensepolicy.
Modular Switches
Both MPUs and LPUs have their ownCPUs. Local attack defense policies are configured differentially for MPUs andLPUs.
Before creating and applying attackdefense policies, check attack information on the MPUs and LPUs. If the attackinformation on the MPUs and LPUs is consistent, apply the same attack defensepolicy to the MPUs and LPUs; otherwise, apply different policies to them.
i.         Apply an attack defense policyto MPU.
<HUAWEI> system-view
[HUAWEI] cpu-defend-policy policy1 
[HUAWEI] quit
ii.       Apply an attack defense policyto an LPU.

If an attack defense policy has beenapplied to all LPUs, it cannot be applied to the specified LPU. In a similarmanner, if an attack defense policy has been applied to a specified LPU, itcannot be applied to all LPUs.
   If all LPUs process similar services, apply an attack defense policyto all LPUs.
<HUAWEI> system-view
[HUAWEI] cpu-defend-policy policy2global
   If LPUs process different services, apply an attack defense policyto the specified LPU.
<HUAWEI> system-view
[HUAWEI] slot 1
[HUAWEI-slot-1] cpu-defend-policypolicy2
Fixed Switch
n   Apply an attack defense policy to a stand-alone switch.
<HUAWEI> system-view
[HUAWEI] cpu-defend-policy policy1global
n   In a stack:
- Apply an attack defensepolicy to the master switch.
<HUAWEI> system-view
[HUAWEI] cpu-defend-policy policy1
- Apply an attack defensepolicy to all stacked switches.
<HUAWEI> system-view
[HUAWEI] cpu-defend-policy policy1global
c.        View attack source information.
After configuring local attackdefense based on attack source tracing, run the display auto-defend attack-source and display auto-defend attack-source slot slot-id commands to view attack source information.

The MAC address of gateway should beexcluded from the suspicious attack sources.
3.        Select an appropriate methodbased on the attack source information and networking.
          Configure ARP security toprevent ARP attacks.
The switch provides ARP security toprevent ARP and ARP Miss packet attacks.
For details about ARP security, seeARP Security Solutions in section ARP Security Configuration in theConfiguration Guide - Security.
          Configure an attack sourcetracing action: discard attack packets within the specified period.
# Discard attack packets within 300s.
<HUAWEI> system-view
[HUAWEI] cpu-defend policy policy1
[HUAWEI-cpu-defend-policy-policy1] auto-defendenable  //Enable attack sourcetracing. By default, this function is disabled.
[HUAWEI-cpu-defend-policy-policy1] auto-defendaction deny timer 300  //By default,attack source tracing action is not enabled.
          Configure a blacklist for localattack defense. The packets from the users in the blacklist are discarded.
If an attack source is considered asattacker (for example, attack source address is 1.1.1.0/24), add the users withthe specified characteristics to a blacklist.
# Configure ACL 2001 to match thepackets with source address 1.1.1.0/24 and discard the packets matching theACL.
[HUAWEI] acl number 2001
[HUAWEI-acl-basic-2001] rule permitsource 1.1.1.0 0.0.0.255
[HUAWEI-acl-basic-2001] quit
[HUAWEI] cpu-defend policy policy1
[HUAWEI-cpu-defend-policy-policy1] blacklist1 acl 2001
          Configure attack source tracingaction: shut down the interface receiving attack packets.
If attack packets are sent from aspecified interface, and shutdown of this interface does not affect services,use this method.
 


Shutting down aninterface may cause a service interruption and affect valid users. Use thismethod with caution.

# Configure attack source tracingaction: shut down the interface receiving attack packets.
<HUAWEI> system-view
[HUAWEI] cpu-defend policy policy1
[HUAWEI-cpu-defend-policy-policy1] auto-defendenable  //Enable attack sourcetracing. By default, this function is disabled.
[HUAWEI-cpu-defend-policy-policy1] auto-defendaction error-down
After the attack source is located andhandling method is used, check whether packet loss is relieved. If not, see thenext section.
2.8 Checking Whether the Rate of Packets Sent to the CPU Exceeds theLimit

The switch uses different CPCAR values fordifferent types of protocol packets. Generally, the default CPCAR values canmeet service requirements. The CPCAR values of some protocol packets may needto be changed according to service scale and user network environment.
1.        Run the display cpu-defend statistics all command to view statistics on thepackets sent to the CPU, determining whether service packets are lost.
2.        Run the display cpu-defend configuration [ packet-type packet-type ]{ all | slot slot-id | mcu } command to view the rate limitsfor the packets sent to the CPU.
3.        Run the display cpu-defend rate [ packet-typepacket-type ] { all | mcu | slot slot-id } command to view the rate of packets sent to the CPU.
4.        Determine whether the CPCARvalues meet network requirements.
For example, if a network has lostpackets, check whether the CPCAR setting matches the network scale.
# Check whether ICMP packets sent to theCPU has been dropped.
<HUAWEI> display cpu-defend statistics all                                          
 Statistics on mainboard:                                                       
--------------------------------------------------------------------------------
Packet Type         Pass(Packet/Byte)  Drop(Packet/Byte) Last-dropping-time  
--------------------------------------------------------------------------------
icmp                          880928            44380928  2016-08-10 13:09
                               25301              693450
...
A total of 44380928 ICMP packets on theMPU are dropped.
# Check the rate limit on the ICMPpackets sent to the CPU.
<HUAWEI> display cpu-defend configuration packet-type icmp all                       
Car configurations on mainboard.                                                
----------------------------------------------------------------------          
Packet Name         Status     Cir(Kbps)  Cbs(Byte)  Queue  Port-Type          
----------------------------------------------------------------------          
icmp                 Enabled         256       48128      3        NA          
----------------------------------------------------------------------          
...
The rate of ICMP packets that can besent to the CPU is 256 kbit/s = 256*1024 bit/s = 262144 bit/s.
# Check the rate of ICMP packets sent tothe CPU.
<HUAWEI> display cpu-defend rate packet-type icmp all                               
Info: Please wait for a moment....                                              
Cpu-defend rate on mainboard:                                                   
------------------------------------------------------------------------------- 
Packet Type           Pass(bps)    Drop(bps)       Pass(pps)       Drop(pps)    
------------------------------------------------------------------------------- 
icmp                      239211      3389532          5854            69585    
------------------------------------------------------------------------------- 
...
The rate of ICMP packets sent by the MPUto the CPU is 239211 bit/s, which approximates to the maximum value 262144.Many packets are dropped. This indicates that the CPCAR value does not matchthe network scale.
5.        Improper CPCAR settings willaffect network services. Therefore, adjust the CPCAR settings under theguidance of technical support personnel.
2.9 Checking the Related Configurations

1.        Check whether theconfigurations of interfaces, VLANs, VLANIF interfaces, or globalconfigurations cause packet loss.
The configurations to be checkedinclude:
          Check whether trafficfiltering, suppression, or rate limiting is configured and whether the trafficpolicy configuration is appropriate. For example, if the traffic policy is configuredto drop a type of packets, these packets will be dropped.
n   Check the traffic policy configuration. Check whether the trafficpolicy is correctly applied and whether the traffic behavior and trafficclassifier in the traffic policy have configurations leading to packet loss.
# Run the display traffic policy user-defined command to view the trafficpolicy configuration.
<HUAWEI> display traffic policy user-defined
  User Defined Traffic PolicyInformation:                                      
  Policy: p1                                                                    
   Classifier: c1                                                               
    Operator: AND                                                               
     Behavior: b1                                                               
      Permit 
      Remark:                                                                   
        Remark 8021p 4                                                          
      Committed Access Rate:                                                    
        CIR 10000 (Kbps), PIR 10000(Kbps), CBS 1880000 (byte), PBS 3130000 (byte)                                                                              
        Color Mode: color Blind                                                 
        Conform Action: pass                                                    
        Yellow  Action: pass                                                    
        Exceed  Action: discard                                                  
                                                                                
Total policy number is 1
# Run the display traffic behavior user-defined[ behavior-name ]command to check whether the traffic behavior has configurations leading topacket loss.
<HUAWEI> display traffic behavior user-defined
  User Defined Behavior Information:                                            
    Behavior: b1
      permit
      Remark:                                                                    
        Remark 8021p 4                                                          
      Redirect:                                                                 
        Redirect cpu                                                             
      Committed Access Rate:                                                    
        CIR 10000 (Kbps), PIR 10000(Kbps), CBS 1880000 (byte), PBS 3130000 (byte)
        Color Mode: color Blind                                                  
        Conform Action: pass                                                    
        Yellow  Action: pass                                                    
        Exceed  Action: discard                                                  
                                                                                
Total behavior number is 1    


# Run the display traffic classifier user-defined [ classifier-name ] command to check the traffic classifierconfiguration.
<HUAWEI> display traffic classifier user-defined
  User Defined Classifier Information:
   Classifier: c1
    Precedence: 5
    Operator: AND
    Rule(s) : if-match acl 2046
             
Total classifier number is 1
# Run the display acl { acl-number |all } command to check whether theACL contains the deny rule.
<HUAWEI> display acl all                                    
 Total nonempty ACL number is 1                                                 
                                                                                
Advanced ACL 2046, 1 rule                                                       
Acl's step is 5                                                                 
 rule 5 deny icmp                                             
If the configurations areincorrect, modify the configurations.
n   Check traffic suppression configuration: When a broadcast packetattack occurs or you need to suppress broadcast packets for other reasons, runthe broadcast-suppression command.
broadcast-suppression: sets the maximumtraffic rate of broadcast packets that can pass through an interface. When therate of broadcast packets reaching the interface exceeds the threshold, theexcess packets are discarded. No log is recorded for packet loss.
2.        Check whether security featuresare configured, such as port security, IPSG, and URPF.
port-securityenable: After port security is enabled on aninterface, MAC addresses learned by the interface change to secure dynamic MACaddresses. When the maximum number of secure dynamic MAC addresses learned onan interface reaches the limit (the value is 1 by default), the interface doesnot learn new MAC addresses. It discards packets with new source MAC addresses.
ipsource check user-bind enable: enables IP packetcheck, including IP address, MAC address, VLAN, and interface information inthe IP packets.
urpfstrict: enables URPF strict check on the interface.The packets received by subinterfaces cannot pass strict check and will bedropped.
If the BPDU function is disabled on theinterface, BPDUs cannot be transparently transmitted, and will be dropped.
Generally, the destination MAC addressof BPDUs is 01:80:C2:00:00:xx. By default, an inbound interface will dropreceived BPDUs. To transparent transmit BPDUs, run the bpdu bridge enable command in the interface view.
3.        If the switch performs Layer 2forwarding, perform the following checks:
a.        Check that the VLANconfiguration on the interface is correct.
n   The interface is not added to a specified VLAN; therefore, it doesnot allow packets from the VLAN to pass through.
Run the display vlanvlan-idcommand to check whether the interface is added to any VLAN in untagged ortagged mode. If PVID is configured on the interface, the PVID is added tountagged packets on the inbound interface. You need to add the interface toPVID VLAN.
<HUAWEI> display vlan 10                                                       
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;               
MP: Vlan-mapping;               ST:Vlan-stacking;                              
#: ProtocolTransparent-vlan;    *:Management-vlan;                             
--------------------------------------------------------------------------------
                                                                                
VID  Type    Ports                                                              
--------------------------------------------------------------------------------
10   common  UT:GE2/9/0/1(D)    GE2/9/0/3(D)                                    
             TG:GE2/9/0/4(D)                                                    
                                                                                
VID  Status  Property     MAC-LRN Statistics Description                       
--------------------------------------------------------------------------------
10   enable  default      enable  disable    VLAN 0010
n   Check whether the inbound and outbound interfaces belong to the sameservice VLAN.
n   The interface is configured to drop incoming tagged packets.
port discard tagged-packet: configuresthe interface to drop incoming tagged packets. If the packet loss is caused byincorrect configuration, run the undoport discard tagged-packet command to modify the configuration.
n   The interface is configured to drop the packets that do not matchany selective QinQ or VLAN mapping entry.
qinq vlan-translation miss-drop:configures the interface configured with selective QinQ and VLAN mapping todrop the received packets that do not match any selective QinQ or VLAN mappingentry. If the packet loss is caused by incorrect configuration, run the undo qinq vlan-translation miss-dropcommand to modify the configuration.
If selective QinQ isconfigured on the interface, add the interface to the VLAN specified by theouter VLAN tag that replaces the original outer VLAN tag of the packets.
b.        Check that the MAC address ofLayer 2 packets is learned correctly.
Run the display mac-address mac-addresscommand to check whether the bindings between the MAC address, VLAN, andinterface are correct.
<HUAWEI> display mac-address
------------------------------------------------------------------------------- 
MAC Address          VLAN/VSI                    Learned-From        Type       
-------------------------------------------------------------------------------
0022-0022-0033       100/-                       GE1/0/1             dynamic 
0000-0000-0001       -/HUAWEI                    GE1/0/2             static 
-------------------------------------------------------------------------------
Total items displayed = 2
If the source MAC address is notlearned, reconfigure the bindings between the MAC address, VLAN ID, andinterface number.
If selective QinQ is configured on aninterface, the source MAC addresses of interfaces in the VLAN specified by thereplaced outer VLAN tag of the packets are learned.
c.        Check whether any MAC addressconfiguration causes packet loss.
n   MAC address learning is disabled on the interface and the interfaceis configured to drop packets with unknown source MAC addresses.
Check configurations in theinterface or VLAN view. If the command output includes mac-address learning disable action discard, the switch does notlearn MAC addresses on the interface. If a matching entry is found in the MACaddress table, the packets are forwarded based on the MAC address table.Otherwise, the packets are dropped.
n   Check whether MAC address limiting rules are configured.
Check configurations in theinterface or VLAN view. If the command output includes mac-limit maximum max-num,the packets of which the source MAC addresses are not included in the MACaddress table are dropped after the number of MAC address entries reaches thelimit.
n   Check whether a static MAC address is configured.
Run the display mac-address static command to view static MAC addressentries.
If a static MAC address isconfigured, only the interface bound to the static MAC address processes thepackets with this MAC address. Other interfaces drop the packets with this MACaddress.
n   Check whether a blackhole MAC address is configured.
Run the display mac-address blackhole command to view blackhole MAC addressentries.
When a blackhole MAC addressis configured, the system drops a packet if the source or destination MACaddress of the packet is the blackhole MAC address.
4.        If the switch performs Layer 3forwarding, perform the following checks:
a.        Check whether the ARP entriesexist, conflict, or the number of ARP entries exceeds the limit.
n   Run the display arp allcommand to check whether the local end has learned the ARP entry from theremote end.
n   Run the display arp learningstrict command to check whether ARP strict learning is configured globallyor on a VLANIF interface.
<HUAWEI> display arp learning strict
The global configuration:arp learning strict
 Interface                           LearningStrictState
------------------------------------------------------------
 Vlanif100                           force-disable
 Vlanif200                           force-enable
------------------------------------------------------------
 Total:2
 Force-enable:1
 Force-disable:1
After this function isconfigured, the switch learns only the ARP Reply packets in response to the ARPRequest packets sent by itself.
n   Run the display logbuffercommand on a switch to view logs. If the following information is displayed,the switch received ARP packets with conflicting IP addresses.
ARP/4/ARP_DUPLICATE_IPADDR:Receivedan ARP packet with a duplicate IP address from the interface.(IpAddress=[IPADDR], InterfaceName=[STRING], MacAddress=[STRING])
Handling method:
Run the arp anti-attack gateway-duplicate enable command in the system viewto enable ARP gateway anti-collision function. After receiving addresscollision packets, the switch delivers attack defense entries and drops thepackets from the same source MAC address within a certain period (the attacksource cannot access the network within this period).
If permitted by the customer,you can configure a local attack defense blacklist to filter packets, andconfigure a blackhole MAC address entry to prevent the attacker from accessingthe network.
For details about theconfiguration, see 2.7Checking Attacks.
b.        Check that a reachable routeexists between the local and remote ends.
Run the display ip routing-table and displayfib commands to check the routes along the forwarding path. Check whetherthe local end has a reachable route to the remote end, and whether the remoteend has a return route to the local end. Ensure that the routing protocol isproperly configured.
Run the display ip routing-table statistics command to check whether theexcessive routing entries cause a failure to deliver hardware entries.
c.        For a Layer 3 subinterface,check whether ARP broadcast termination (arpbroadcast enable) is configured. Then the subinterface will drop thereceived broadcast packets.
5.        Check whether the localinterface is blocked by STP, RRPP, LDT, or Smart Link.
Generally, an interface cannot runmultiple ring protocols. If a ring protocol is configured on the interface,check the protocol type and the interface status. STP and RRPP are used asexamples.
          If STP is configured, checkwhether the interface is blocked by STP.
Run the display stp brief command to view interface status. In normalsituations, the STP State fieldvalue is FORWARDING.
<HUAWEI> display stp brief
 MSTID  Port                        Role  STP State    Protection 
    0   GigabitEthernet1/0/1       DESI  FORWARDING      NONE     
    0   GigabitEthernet1/0/2       DESI  FORWARDING      NONE     
    0   GigabitEthernet1/0/4       ROOT  FORWARDING      NONE 
If the value of STP state isDISCARDING, the interface is blocked. Run the stp priority priority-levelcommand in the system view to set the STP priority. Ensure that the local portis selected as the root bridge and will not be blocked. (The value of priority-level ranges from 0 to 61440. Asmall value indicates a high priority. A low priority can make a port becomethe root bridge.)
          If RRPP is configured, checkwhether the interface is blocked by RRPP.
Run the display rrpp verbose domain domain-indexcommand to view interface status. In normal situations, the Port status field is Up.
<HUAWEI> display rrpp verbose domain 1
Domain Index   : 1
Control VLAN   : major 400    sub 401
Protected VLAN : Reference Instance 30
Hello Timer    : 1 sec(default is 1sec)  Fail Timer : 6 sec(default is 6sec)
RRPP Ring      : 1
Ring Level     : 0
Node Mode      : Master
Ring State     : Complete
Is Enabled     : Enable                        Is Active : Yes
Primary port   :GigabitEthernet1/0/1          Port status:UP
Secondary port : GigabitEthernet1/0/2         Port status: BLOCKED
If the value is BLOCK, the port is blocked. A secondary interface is blocked byRRPP. You need to modify the configuration so that the interface is not thesecondary interface.
2.10 Determining Packet Loss Location Based on Traffic Statistics

1.        Apply a traffic policy to theinbound interface and outbound interface of the device where packet lossoccurs. Collect statistics on packets of specified type in the inbound andoutbound interfaces and check whether these packets are discarded on the localdevice.
In this example, traffic policy isconfigured on switch_3, switch_2, and switch_1. View traffic statistics fromport a to port f.
Figure 2-3 Deploying the traffic policy

 
Configuretraffic statistics collection.
# Configure traffic statisticscollection in the inbound direction of port a on switch_3.
a.        Configure ACL rules.
<Switch_3> system-view
[Switch_3 acl number 3000
[Switch_3-acl-adv-3000] rule permit icmpsource 192.168.100.1 0 destination 202.10.1.1 0
[Switch_3-acl-adv-3000] quit
b.        Configure a traffic classifier.
[Switch_3] traffic classifier 3000
[Switch_3-classifier-3000] if-match acl3000
[Switch_3-classifier-3000] quit
c.        Configure a traffic behavior.
[Switch_3] traffic behavior 3000
[Switch_3-behavior-3000] statisticenable
[Switch_3-behavior-3000] quit
d.        Configure a traffic policy.
[Switch_3] traffic policy 3000
[Switch_3-trafficpolicy-3000] classifier3000 behavior 3000
[Switch_3-trafficpolicy-3000] quit
e.        Apply the traffic policy to theinterface.
[Switch_3] interface gigabitethernet 1/0/2   
[Switch_3-GigabitEthernet1/0/2] traffic-policy3000 inbound   
[Switch_3-GigabitEthernet1/0/2] quit
# Configure traffic statisticscollection in the outbound direction of port b on switch_3.
a.        Configure ACL rules.
[Switch_3] acl number 3001
[Switch_3-acl-adv-3001] rule permit icmpsource 202.10.1.1 0 destination 192.168.100.1 0
[Switch_3-acl-adv-3001] quit
b.        Configure a traffic classifier.
[Switch_3] traffic classifier 3001
[Switch_3-classifier-3001] if-match acl3001
[Switch_3-classifier-3001] quit
c.        Configure a traffic behavior.
[Switch_3] traffic behavior 3001
[Switch_3-behavior-3001] statisticenable
[Switch_3-behavior-3001] quit
d.        Configure a traffic policy.
[Switch_3] traffic policy 3001
[Switch_3-trafficpolicy-3001] classifier3001 behavior 3001
[Switch_3-trafficpolicy-3001] quit
e.        Apply the traffic policy to theinterface.
[Switch_3] interface gigabitethernet 1/0/2   
[Switch_3-GigabitEthernet1/0/2] traffic-policy3001 outbound   
[Switch_3-GigabitEthernet1/0/2] quit
Configure traffic statistics collectionon switch_2 and switch_1 similarly. For the traffic statistics collectionconcepts and configurations, see 5.1 Configuring Traffic Statistics Collection.
Analyzethe statistics.
a.        Run the display traffic policy statistics interface interface-type interface-numberinbound/outbound verbose rule-base command to view traffic statistics onthe interface.
b.        Compare the Passed counts inthe inbound direction of port a and outbound direction of port b, outbounddirection of port b and inbound direction of port c, inbound direction of portc and outbound direction of port d, outbound direction of port d and inbounddirection of port e, and inbound direction of port e and outbound direction ofport f. Determine the packet loss location.
In this example, compare the Passedcounts in the inbound direction of port a and outbound direction of port b, andin the outbound direction of port b and inbound direction of port c.
n   If the Passed count in the inbound direction of port a approximateto the Passed count in the outbound direction of port b, no packet is dropped.
n   If the Passed count in the inbound direction of port a is greaterthan the Passed count in the outbound direction of port b, packets are droppedon switch_3.
Locate the problem accordingto 2.10.1 Loading the Latest Patch Matching the System Software Version and 2.10.2 Resetting or Replacing the Problematic Cards.
n   If the Passed count in the outbound direction of port b approximateto the Passed count in the inbound direction of port c, no packet is dropped.
n   If the Passed count in the outbound direction of port b is greaterthan the Passed count in the inbound direction of port c, packets are droppedon the link between switch_3 and switch_2. Locate the problem according to 2.10.3 Checking the Physical Links.
2.10.1 Loading the Latest Patch Matching the System Software Version

Load and activate the latest patch matchingthe system software version. Then check whether the problem is fixed.

Visit http://support.huawei.com/enterprise/http://support.huawei.com/ toobtain the corresponding patch file and documents (patch release notes andinstallation guide).
2.10.2 Resetting or Replacing the Problematic Cards

Reset or remove the cards and check whetherpacket loss is resolved. Ensure that this operation does not affect services.Contact the technical support engineers to replace cards.
2.10.3 Checking the Physical Links

In this example, view traffic statistics onswitch A and switch B. You can see that the number of incoming packets onswitch B is smaller than the number of outgoing packets on switch A, but thenumber of incoming packets on switch A is smaller than the outgoing packets onswitch B, indicating that the ping packets have been dropped on the physicallink between switch A and switch B.
Common physical link faults are as follows:
l  Cable connectors are connectedtoo loosely.
l  The optical module wavelengthsdo not meet actual needs.
l  Communication interfaces aredamaged.
l  Cable is too long or damaged.
Locate the physical link faults as follows:
1.        Observe the indicator status.
If the indicator is off, the port is notconnected. Replace the port or network cable and try again.
2.        Check whether the cables andhardware modules are properly connected.
If the cables and hardware modules areproperly connected but the fault persists, go to the next step.
3.        Check whether the links andinterface modules work properly.
          If the two devices areconnected by a twisted pair, check the items according to Table 2-3.
Table 2-3 Check on the devices connected by a twisted pair

   Item
   
   Expected Result
   
   Follow-up Procedure
   
  Twisted pair working status
  
  The tester shows that the twisted pair  works properly.
  
  If the twisted pair is faulty, replace  it.
  
  Twisted pair length
  
  The twisted pair length does not exceed  100 m.
  NOTE
  The 10/100/1000M electrical  interfaces use RJ45 connectors and Category 5 or higher twisted pairs. The  maximum transmission distance of such cables is 100 m.
  
  If the cable length is longer than 100 m,
  l   Shorten the distance between  the two devices.
  l   If the distance between the  two devices cannot be shortened, deploy a repeater or switch between the  devices.
  
  Twisted pair type
  
  A straight-through cable connects  Ethernet interfaces between the following devices:
  l   Router and hub
  l   Router and switch
  l   PC and switch
  l   PC and hub
  A crossover cable connects Ethernet  interfaces between the following devices:
  l   Two routers
  l   Router and PC
  l   Two hubs
  l   A hub and a switch
  l   Two switches
  l   PCs
  
  If the twisted pair type is incorrect,  replace the twisted pair.
  
  Local and remote electrical modules
  
  1.     Check whether the metal pins  of the port are bent or deviate.
  2.     Check whether the cable is  loose or damaged.
  
  If the electrical modules work  abnormally, replace them.
  

 
l  If the two devices areconnected by optical fibers, check the items according to Table 2-4.
Table 2-4 Check on the devices connected by a fiber

   Item
   
   Expected Result
   
   Follow-up Procedure
   
  Types of optical modules and optical  fibers
  
  Verify that the fiber type is correct.
  
  If the optical fiber type does not match  the optical module type, replace the optical module or optical fiber.
  
  Optical fiber length and maximum  transmission distance of optical modules
  
  The optical fiber length must be shorter  than the maximum transmission distance of an optical module.
  
  If the optical fiber length exceeds the  maximum transmission distance of the optical modules, shorten the distance  between the devices or use optical modules with a longer transmission  distance.
  
  Optical signal attenuation
  
  The optical signal attenuation range  depends on the optical module type.
  
  1.     If the attenuation is too  large, replace the fiber.
  2.     If the attenuation is still  large, shorten the fiber.
  
  Optical fiber working status
  
  l   The tester shows that optical  signals are sent and received properly.
  l   In a loopback test, the two  interfaces are Up.
  
  1.     If the fiber is faulty,  replace it.
  2.     If the fault persists,  replace the optical modules.
  

 
Look up parameters of the optical moduleaccording to section "Pluggable Modules for Interfaces" in theHardware Description. The parameters include the center wavelength,transmission distance, fiber types supported, receive optical power, andtransmit optical power.
If the optical power is abnormal, an alarmis reported. You can also view the alarm information to know whether opticalpower is normal.
2.11 Contacting Technical Support Personnel

If the fault persists, collect networkinginformation (including port connections), problematic source IP address, sourceMAC address, destination IP address, destination MAC address, inbound port,outbound port, and logs (read the log.logfile or run the display logbuffer command).Provide the collected information to technical support engineers.


3 Typical Cases


3.1  A Few Packets Are Dropped 2 Minutes AfterMulticast Service Is Enabled
3.2  Management User Cannot Telnet to a Switch
3.3  Ping Packets from Monitoring Server toServers Are Dropped
3.4  Inbound Packets Are Dropped
3.5  Pixelation Appears on the Screen When aSwitch Is Connected to Two Receivers
3.6  User Hosts Go Online Slowly and Pinging theGateway Has a Delay
3.7  Packets Are Dropped When a Switch IsConnected to a Third-Party Device
3.8  Packets Are Lost When Access Users Ping aStacked Switch
3.9  Users Connected to a Switch Cannot Access theNetwork
3.10  Packets Are Lost When Users Access a Server
3.11  Artifacts Occur When Users Watch Videos
3.12  Pixelization Occurs When Users Watch Videos
3.13  Pixelization Occurs When Users Watch IPTVDuring Peak Hours
3.14  Image Pauses Occur When Users Watch Videos
3.1 A Few Packets Are Dropped 2 Minutes After Multicast Service IsEnabled

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
As shown in the figure, a switch isupstream to a multicast server and downstream to user hosts. The switch hasmany multicast entries. Test the multicast forwarding performance on theswitch.

FaultSymptom
When the interface GE1/0/1 that receivesmulticast packets and the interface GE1/0/2 that sends multicast packets are inthe same VLAN, a few packets are dropped 2 minutes after the multicast entriesare created, and then the packets of multicast groups are randomly dropped.
If the two interfaces are added todifferent VLANs, no packet is lost.
CauseAnalysis
There are multiple IGMP queriers withdifferent query intervals on the network. The inconsistent query intervalscause incorrect aging of Layer 2 multicast forwarding entries.
TroubleshootingProcedure
1.        Run the display cpu-defend statistics all command to check statistics onthe packet sent to the CPU. The statistics show that no IGMP packet is dropped.
2.        Run the display multicast forwarding-table command to check Layer 3multicast forwarding entries. In the command output, the up-time value of theentry is 39 minutes, indicating that the entry has been aged out before.
3.        Run the display igmp-snooping port-info verbose command to check Layer 2multicast forwarding entries. The outbound interfaces have different up-timevalues, some of which are very short. The expire-time of most outboundinterfaces is updated when the up-time approximates to 5 minutes.
4.        Run the display igmp interface command. The command output shows that theIGMP querier is located on the tester, but not the switch. On the tester, thequery interval is set to 125 seconds.
The default IGMP snooping query intervalon the switch is 60 seconds, so the Layer 2 multicast entry aging time is 130seconds. This means that the switch has only 5 seconds to update entries. Theswitch cannot process all the received messages in such a short time, so someentries are aged out, although they should not be aged out. As a result,packets of some groups are dropped.
Solution
Disable the querier function on the tester.
Conclusion
Random packet loss is usually caused by anyof the following:
l  Some IGMP protocol packets aredropped due to CPCAR exceeding, so some multicast forwarding entries are agedout incorrectly.
l  There are multiple querierswith different query intervals on the shared network segment, so some multicastforwarding entries are aged out incorrectly.
l  The burst traffic rate on aninterface exceeds the buffer capacity, so packets are dropped randomly on theinterface.
When multicast packet loss occurs, excludethe preceding causes one by one to determine the root cause. After that, adjustthe network deployment to solve the problem.
3.2 Management User Cannot Telnet to a Switch

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
The administrator's terminal is connectedto a switch.
FaultSymptom
A switch has a high CPU usage, users cannotlog in to the switch through Telnet or console port, and the switch prints alarge number of Telnet login failure logs.
CauseAnalysis
When a switch is attacked by Telnet and FTPpackets, the CPU usage is high and users cannot log in to the switch.
TroubleshootingProcedure
1.        Clear statistics on the Telnetpackets sent to the CPU.
<Quidway> reset cpu-defend statistics packet-type telnet all
2.        Wait for one minute and viewstatistics about the Telnet packets sent to the CPU again.
If many Telnet packets are dropped, aTelnet attack may occur.
[Quidway] display cpu-defend statistics packet-type telnet slot 2
 Statistics on slot 2:
----------------------------------------------------------------------
Packet Type         Pass(Bytes)  Drop(Bytes)  Pass(Packets)   Drop(Packets)
----------------------------------------------------------------------
Telnet              40800      3576800      600           52600  //Many packets are discarded.
----------------------------------------------------------------------
3.        Dropping Telnet packets is notrecommended. You are advised to:
          Check whether the IP addressessending many Telnet packets are authorized.
For example, if the IP address of thedirectly connected gateway sends many Telnet packets, these packets cannot bedropped.
If the attack source address is anunauthorized address, configure traffic policy to drop the packets for a fixedswitch or configure a blacklist or configure attack source tracing for amodular switch.
# Configure attack source tracing toidentify the Telnet attack source and configure automatic defense function.
#
cpu-defend policy 1
 auto-defend enable
 auto-defend attack-packet sample 5
 auto-defend threshold 30  //Set the attack identification threshold to30 pps.
 auto-defend trace-type source-macsource-ip
 auto-defend protocol telnet
 auto-defend action deny timer 30  //Set the punishment interval to 30s.
#
<Quidway> display auto-defend attack-source slot 2detail
    Attack Source IP Table (MPU): 
  --------------------------------------------------------
    IP address                     10.1.0.2  //Exclude the gateway's IP address.
      TELNET:                    2873 
      Total                        2873 
 ----------------------------------------------------------
   Total: 1
          To ensure security for theTelnet control layer, configure an ACL on the user-interface vty interface andconfigure AAA to control Telnet users' right. Only the specified IP addressesare allowed.
3.3 Ping Packets from Monitoring Server to Servers Are Dropped

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
Switch_1 and switch_2 function as gatewaysto connect to the monitoring server. Switch_3 is a Layer 2 access device thatconnects to server A and server B.

FaultSymptom
Ping tests are performed on the monitoringserver to test interoperability with the server A and server B. Severe packetloss occurs.
CauseAnalysis
1.        When the monitoring serveraccesses the servers, Switch_1 and switch_2 do not have ARP entries of theservers. This triggers a large number of ARP Miss packets.
2.        On switch_1 and switch_2, ratelimiting for the ARP Miss messages from the monitoring server 10.0.0.1 is set.
arp-miss speed-limit source-ip 10.0.0.1 maximum 100  //Set the rate limit for ARPMiss messages from source address 10.0.0.1 to 100 pps.
3.        When a switch generates manyARP Miss messages from the same source IP address, excessive packets aredropped.
TroubleshootingProcedure
1.        Analyze the ARP and securityconfiguration on the device.
2.        Check the ARP information ofthe device and run the display arppacket statistics command to check whether ARP packets are discarded.
3.        Run the display arp anti-attack configuration all command to check the ARPanti-attack configuration. Check whether ARP Miss rate limit and ARP rate limithave been configured.
Solution
1.        Disable rate limiting for ARPMiss messages.
2.        Set the aging time of ARPentries to 1200s (default value).
Conclusion
When locating ping packet loss problems,check the security-related configurations. The security commands, including arp-miss speed-limit source-ip, arp anti-attack rate-limit, and icmp rate-limit enable should beconfigured based on actual situations.
3.4 Inbound Packets Are Dropped

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
Switch_1 is downstream to a packet analysisserver, and upstream to a splitter, which is connected to switch_2 and switch_3.

FaultSymptom
Packets from switch_2 and switch_3 aredropped on GE0/0/1 of switch_1.
CauseAnalysis
Among the traffic forwarded by the upstreamdistribution device:
l  The source MAC address and VLANID in the traffic from switch_2 are the same as the destination MAC address andVLAN ID in the traffic from SwitchC.
l  The source MAC address and VLANID in the traffic from switch_3 are the same as the destination MAC address andVLAN ID in the traffic from switch_2.
Packets cannot be forwarded because the inboundport learns the same source and destination MAC addresses of packets.
TroubleshootingProcedure
1.        Check whether traffic in twodirections (the source and destination MAC addresses in two directions arereverse) is forwarded by the same interface based on services.
2.        Obtain packets and checkwhether an interface has learned the same source and destination MAC addresses.
3.        Disable MAC address learning onthe inbound interface or in a VLAN.
Conclusion
Packets cannot be forwarded because aninbound interface has learned the same source and destination MAC addresses ofpackets.
3.5 Pixelation Appears on the Screen When a Switch Is Connected to TwoReceivers

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
A multicast source is directly connected tothe switch. The switch has Layer 3 multicast service configured and connects toa receiver network segment through a VLANIF interface. The two receivers aremonitoring terminals that monitor multicast program information.

FaultSymptom
The customer finds frequent pixelation onthe monitoring terminals.
CauseAnalysis
Only Layer 3 multicast is configured on theswitch, so the outbound interface in multicast forwarding entries is the VLANIFinterface. However, only Layer 3 multicast on a switch cannot implement preciseforwarding. That is, multicast traffic will be forwarded to all physicalinterfaces in the VLAN matching the outbound VLANIF interface.
The two terminals are connected to theswitch through one VLAN, so the second terminal can receive the programs (whichis not desired) monitored by the first terminal. Similarly, the first terminalcan receive the programs monitored by the second terminal.
Limited by their packet processingcapacity, the terminals cannot process all the received packets in a timelymanner. Consequently, pixelation occurs.
TroubleshootingProcedure
Check whether igmp-snooping enable is configured in the view of the VLAN matchingthe VLANIF interface where Layer 3 multicast is configured.
Solution
l  Replace Layer 3 multicast withLayer 2 multicast to implement accurate forwarding of multicast data traffic.
l  If the customer requires Layer3 multicast, configure both Layer 2 and Layer 3 multicast on the switch.
Conclusion
Pixelation is often caused by packet lossor duplicate packets.
Common causes for packet loss include:
l  Packets are dropped due tocongestion on the outbound interface.
Check whether there are Discard counts.
          Install the latest patch forthe system software version in use, and check whether the problem is resolved.Obtain the software version, patch version, product model, and configuration ofthe switches to determine whether the problem is caused by a known issue.
          Adjust the mode in which themulticast server sends packets to relieve traffic congestion.
          If multiple traffic flows aretransmitted on the same link, increase the link bandwidth between networkdevices.
l  Rate limiting is configured onthe outbound interface, and the rate of incoming packets exceeds the rate limitduring peak hours.
Common causes for duplicate packetsinclude:
l  The Layer 3 multicast functioncannot implement accurate forwarding of multicast traffic without the IGMPsnooping function.
l  Multiple multicast sources sentthe same multicast data stream. When this occurs, modify the network deploymentor require receivers to specify a source when requesting programs.
3.6 User Hosts Go Online Slowly and Pinging the Gateway Has a Delay

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
Switch_1 and switch_2 are access switches,which are upstream to the core switch switch_3 and downstream to user hosts.

FaultSymptom
User host A is disconnected from thenetwork. When the user hosts ping the gateway, the gateway responds after adelay. Switch_1 is frequently out of management. Services on switch_2 arenormal and user host B can successfully ping the gateway.
CauseAnalysis
Switch_1 receives ARP packets with fixedsource MAC address. User host A cannot send or receive ARP packets.
TroubleshootingProcedure
Perform the following operations onswitch_1:
1.        Check whether the CPU usage ishigh.
<HUAWEI>display cpu-usage
CPU Usage Stat. Cycle: 10 (Second)
CPU Usage         : 82% Max: 99%
CPU Usage Stat. Time : 2010-12-18 15:35:56
CPU utilization for five seconds: 68%: one minute: 60%: five minutes: 55%.
The CPU usage reaches 82%.
2.        View temporary ARP entries tocheck whether ARP learning is normal.
<HUAWEI>display arp
IP ADDRESS  MAC ADDRESS EXPIRE(M) TYPEVPN-INSTANCE   INTERFACE
VLAN/CEVLAN
------------------------------------------------------------------------------------------------------
10.137.222.139  00e0-fc01-4422            I -         Eth0/0/0
10.137.222.1    0025-9e36-e8c1  20       D-0         Eth0/0/0
10.137.222.100  0025-9e80-b278  6         D-0         Eth0/0/0
10.137.222.99   00e0-4c77-b0e1  9        D-0         Eth0/0/0
10.137.222.173  000f-3d80-cba4  18       D-0         Eth0/0/0
10.137.222.34   0025-9e36-e8c1  1        D-0         Eth0/0/0
10.137.222.172  0016-ec71-ea8c  7         D-0         Eth0/0/0
10.137.222.35   0025-9e36-e8c1  18       D-0         Eth0/0/0
10.137.222.179  0014-2ae2-3128  20       D-0         Eth0/0/0
10.137.222.38   0025-9e36-e8c1  17       D-0         Eth0/0/0
10.137.222.175  0014-2261-2b22  1         D-0         Eth0/0/0
50.1.1.3        Incomplete      1         D-0         GE5/0/0
500/-
50.1.1.2        Incomplete      1         D-0         GE5/0/0
500/-
6.1.1.2         00e0-fc01-4422            I -         Vlanif6
10.0.0.139      00e0-fc01-4422            I -         Vlanif10
192.0.0.4       00e0-fc01-4422            I -         Vlanif192
20.1.1.1        00e0-fc01-4422            I -         Vlanif200
192.168.2.2     00e0-fc01-4422            I -         Vlanif100
------------------------------------------------------------------------------------------------------
Total:16        Dynamic:10      Static:0    Interface:6
The MACADDRESS fields of two ARP entries are Incomplete,indicating temporary entries. Some ARP entries cannot be learned.
3.        Check whether the switch issuffering an ARP attack.
# View statistics about ARP requestpackets sent to the CPU.
<HUAWEI>display cpu-defend arp-request statistics all
Statistics on mainboard:
------------------------------------------------------------------------------------------------------------------
Packet Type         Pass(Bytes)       Drop(Bytes)   Pass(Packets)     Drop(Packets)
-----------------------------------------------------------------------------------------------------------------
arp-request            67908288            0         1061067               0
------------------------------------------------------------------------------------------------------------------
Statistics on slot 4:
------------------------------------------------------------------------------------------------------------------
Packet Type         Pass(Bytes)       Drop(Bytes)   Pass(Packets)     Drop(Packets)
------------------------------------------------------------------------------------------------------------------
arp-request            80928            44380928          2301         693450
------------------------------------------------------------------------------------------------------------------
Statistics on slot 5:
------------------------------------------------------------------------------------------------------------------
Packet Type         Pass(Bytes)       Drop(Bytes)   Pass(Packets)     Drop(Packets)
------------------------------------------------------------------------------------------------------------------
arp-request                 N/A          N/A               0               0
------------------------------------------------------------------------------------------------------------------
Statistics on slot 6:
------------------------------------------------------------------------------------------------------------------
Packet Type         Pass(Bytes)       Drop(Bytes)   Pass(Packets)     Drop(Packets)
------------------------------------------------------------------------------------------------------------------
arp-request                 N/A          N/A               0               0
------------------------------------------------------------------------------------------------------------------
There are a large number of ARP requestpackets on the board in slot 4.
# Configure attack source tracing toidentify the attack source.
<HUAWEI>system-view
[HUAWEI]cpu-defend policy policy1
[HUAWEI-cpu-defend-policy-policy1]auto-defendenable
[HUAWEI-cpu-defend-policy-policy1]auto-defendattack-packet sample 5  //One packetis sampled when every five packets are sent. A small sampling rate will consumemany CPU resources.
[HUAWEI-cpu-defend-policy-policy1]auto-defendthreshold 30  //The packets of whichthe rate reaches 30 pps are considered attack packets. If there are many attacksources, reduce this value.
[HUAWEI-cpu-defend-policy-policy1]undoauto-defend trace-type source-ip source-portvlan  //Identify the attack source based on sourceMAC address.
[HUAWEI-cpu-defend-policy-policy1]undoauto-defend protocol 8021x dhcp icmp igmp tcp telnet ttl-expired udp  //Identify the attack source of the ARPattack.
[HUAWEI-cpu-defend-policy-policy1]quit
[HUAWEI]cpu-defend-policy policy1
[HUAWEI]cpu-defend-policy policy1 global
# View attack source information.
[HUAWEI]display auto-defend attack-source
Attack Source User Table (MPU):
------------------------------------------------------------------------------------------------
MacAddress       InterfaceName      Vlan:Outer/Inner      TOTAL
------------------------------------------------------------------------------------------------
0000-0000-00db  GigabitEthernet2/0/22        193           416
------------------------------------------------------------------------------------------------
The MAC address of attack source is0000-0000-00db, which is connected to GigabitEthernet2/0/22.
If the MAC address has a matching ARPentry, run the display arp | include0000-0000-00db command to check its IP address.
# Choose one method to block the attacksource.
          Configure a blacklist.
#
acl number 4000
 rule 10 permit type 0806 ffff source-mac0000-0000-00db ffff-ffff-ffff
#
cpu-defend policy 1
 blacklist 1 acl 4000  //Add the users with specifiedcharacteristics to the blacklist through an ACL. The switch discards thepackets from the users in blacklist.
#
cpu-defend-policy 1 
cpu-defend-policy 1 global
#
          Configure the attack sourcetracing action.

cpu-defend policy policy1 
 auto-defend enable 
 auto-defend threshold 30 
 undo auto-defend trace-type source-ipsource-portvlan 
 undo auto-defend protocol 8021x dhcpicmp igmp tcp telnet ttl-expired udp
 auto-defend action deny  //Set the attack source tracing action. Theswitch drops all attack packets within the default interval, 300s.
#
 cpu-defend-policy policy1 global 
 cpu-defend-policy policy1 
#
# If you cannot determine the attacksource, use either of the following methods:
          Traffic suppression
Set the maximum rate of broadcast andunknown unicast packets to 100 kbit/s and the maximum percentage of multicastpacket rate to 30%.
#
interface GigabitEthernet12/0/12
 unicast-suppression cir 100 cbs 18800
 multicast-suppression 30
 broadcast-suppression cir 100 cbs 18800
#
          Storm control
Perform storm control on broadcastpackets received on GigabitEthernet12/0/12. In the storm detection interval,the switch performs storm control on packets when the rate of receiving packetson an interface is higher than 8000 pps and forwards packets when the rate ofreceiving packets is lower than 5000 pps.
#
interface GigabitEthernet12/0/12
 storm-control broadcast min-rate 5000max-rate 8000
#
View the storm control effect:
<Quidway> display storm-control interfacegigabitethernet 12/0/12
PortName     Type  Rate     Action    Punish-   Trap Log Interval Last-
                   (Min/Max)           Status                      Punish-Time 
------------------------------------------------------------------------------
GE12/0/12 Broadcast    14881000  shutdown shutdown  off  off 180     2010-12-12 
                   /14881000                                       14:30:00
3.7 Packets Are Dropped When a Switch Is Connected to a Third-PartyDevice

ApplicableProducts and Versions
All products and versions
NetworkingDescription
A switch is connected to a third-partydevice.
FaultSymptom
If the inter-frame gap on the third-partydevice is smaller than 12 bytes, packets are dropped. If the inter-frame gap is12 bytes, no packet is dropped.
CauseAnalysis
If the IFG is set to a small value on theremote device, the remote device forwards packets at a higher rate, causingpacket loss on the local switch.
The packet forwarding rate, also calledthroughput, refers to the data forwarding capability on an interface and ismeasured in packet per second (pps). The packet forwarding rate is calculatedbased on the number of 64-byte data packets transmitted in a period. Thelengths of preamble and IFG affect the packet forwarding rate.
TroubleshootingProcedure
Obtain packets to check whether the IFG is12 bytes (96 bits). Change the IFG to 12 on the remote device.
Conclusion
The settings of interconnection parametersshould be the same on local and remote devices.
3.8 Packets Are Lost When Access Users Ping a Stacked Switch

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
In the following figure, SwitchA_1 andSwitchA_2 set up a stack, SwitchA-stack, while SwitchB_1 and SwitchB_2 set up astack SwitchB-stack.
SwitchA_1 is connected to SwitchB_1 usingtwo uplinks, and SwitchA_2 is connected to SwitchB_2 using two uplinks. Thefour uplinks are bound to Eth-Trunk 100, and users are connected toSwitchA-stack.

FaultSymptom
When the shutdown or undo shutdowncommand is used on any link of the Eth-Trunk and a PC pings SwitchB-stack,packets are lost.
CauseAnalysis
l  A loop occurs on the network ofthe downstream device connected to SwitchB-stack, and packets sent bySwitchB-stack are looped back from the downstream device.
l  A loop is formed betweenSwitchB-stack and the downstream device.
TroubleshootingProcedure
1.        Collect traffic statistics onthe Eth-Trunk of SwitchB-stack and Eth-Trunk of the remote device. If there arestatistics on received packets on the local interface and statistics on sentpackets on the remote interface, a loop occurs on the downstream deviceconnected to SwitchB-stack or a loop is formed between SwitchB-stack and thedownstream device.
2.        Check the network topology,find the loop, and adjust the configuration or change the physical link toeliminate the loop.
Conclusion
This problem is caused by the incorrectconfiguration. During network planning, determine the interfaces that need toexchange STP BPDUs with other devices and determine the devices where STP needsto be disabled.
3.9 Users Connected to a Switch Cannot Access the Network

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
Multiple users are connected to a switchthat performs Layer 2 transparent transmission.
FaultSymptom
Some users cannot access the network, andthere are statistics about discarded ARP packets on the switch.
CauseAnalysis
The arpanti-attack gateway-duplicate enable command is configured on the switch,so all passing ARP packets are sent to the CPU. When a large number of ARPpackets are transmitted on the network, some ARP packets are discarded. As aresult, some users cannot access the network.
TroubleshootingProcedure
1.        Run the display cpu-defend statistics all command to check statistics aboutthe ARP packets sent to the CPU. The command output shows that a large numberof ARP packets are discarded.
2.        Check the device configuration,finding that the arp anti-attackgateway-duplicate enable command has been configured on the switch.
Solution
Delete the arp anti-attack gateway-duplicate enable command configuration andthen check whether there are statistics about discarded ARP packets that aresent to the CPU.
3.10 Packets Are Lost When Users Access a Server

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
In the following figure, the access switchSwitch_2 is uplink connected to the core switch Switch_1 and downlink connectedto users, and Switch_1 is connected to a server.

FaultSymptom
When a user accesses the server, severepacket loss occurs and services are interrupted.
CauseAnalysis
The loopbackinternal command is configured on GE0/0/1 of Switch_2, so GE1/0/2 ofSwitch_1 learns the MAC address of the server. As a result, traffic cannot beforwarded.
TroubleshootingProcedure
1.        Run the display trapbuffer command on Switch_1 to check whether MAC addressflapping occurs. If so, locate the interfaces on which MAC address flappingoccurs.
2.        Delete the loopback internal command configuration on Switch_2.
Conclusion
MAC address flapping is one of the mostcommon causes for packet loss during Layer 2 forwarding. If such a problemoccurs, check whether MAC address flapping occurs.
3.11 Artifacts Occur When Users Watch Videos

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
In the following figure, the Switch isconnected to a multicast server and has GE1/0/0 and GE1/0/2 added to Eth-Trunk0 and Eth-Trunk 1 respectively. Users are connected to the Switch. Themulticast service is deployed on the Switch so that users can play videos.

FaultSymptom
Artifacts occur when users watch videos.
CauseAnalysis
Artifacts are often caused by packet loss,duplicate packets, or packet mis-sequencing. Generally, artifacts are caused bypacket loss or duplicate packets, and the probability of packet loss is higher.When artifacts occur, check the traffic statistics on the upstream anddownstream interfaces of the switch to determine whether packet loss hasoccurred. You can also obtain packets to check for packet loss or duplicatepackets.
In this example, the artifacts are causedby packet mis-sequencing. The cause of packet mis-sequencing is: The Eth-Trunkmember ports on the multicast source do not share traffic on a per flow basis,so packets of the same multicast flow do not arrive at the Switch through thesame Eth-Trunk member port. Due to the mis-sequencing, the Switch cannotforward the multicast packets in a correct sequence, causing artifacts on thereceiver terminal.
TroubleshootingProcedure
1.        Configure traffic statisticscollection on the Switch. The statistics show that no packets are discarded onthe Switch.
2.        Obtain packets on thedownstream device of the Switch, finding that packet mis-sequencing occurs. Asa result, the receiver terminal cannot process packets normally.
3.        Check whether each Eth-Trunkmember port receives packets of the same flow.
          Configure port mirroring on allmember ports of the upstream Eth-Trunk to obtain packets. If each member portreceives packets, the packet sequence cannot be ensured when the packets areforwarded by the Switch.
          If packets cannot be obtained,try to shut down member ports of this Eth-Trunk to retain a single Up port. Ifthe problem is solved, this problem occurs because multiple ports of the Switchreceive packets of the same multicast flow.
Solution
1.        Shut down member ports of theupstream Eth-Trunk and retain a single Up member port (if the port bandwidth issufficient for traffic forwarding).
2.        Fix the problem of the upstreammulticast source to ensure that all packets of a flow are sent to the Switchthrough the same physical port.
Conclusion
Huawei switches forward packets throughhardware, so packet mis-sequencing seldom occurs. If it occurs on a switch,check whether the switch has received packets of the same flow from multipleports.
3.12 Pixelization Occurs When Users Watch Videos

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
In the following figure, Switch_1 isconnected to a multicast server and Switch_2, while Switch_2 is connected tousers. Layer 2 multicast and Layer 3 multicast are configured on GE1/0/2 ofSwitch_1, and Layer 2 multicast is configured on GE0/0/1 of Switch_2.

FaultSymptom
Pixelation occurs on user terminals.
Cause Analysis
Video traffic bursts frequently occur onthe multicast server. When the switches receive multicast traffic, they forwardmultiple copies of the traffic. As a result, the bandwidth required to forwardthe traffic exceeds the limit. When the buffer on the switches is used up,packet loss will occur due to congestion, resulting in pixelation on userterminals.
TroubleshootingProcedure
1.        Check multicast forwardingentries on Switch_1 and Switch_2, finding no exceptions.
2.        Configure a traffic policy onSwitch_2 to collect statistics about incoming packets on GE0/0/1 and outgoingpackets on GE0/0/2. The statistics show that some incoming packets on GE0/0/1are discarded.
3.        Configure a traffic policy onSwitch_2 to collect statistics about incoming packets on GE0/0/1 and outgoingpackets on GE0/0/2. The statistics show that some incoming packets on GE0/0/1are discarded.
4.        Run the display interface interface-typeinterface-number command in any view, or run the display this interface command in the interface view. Check the numberof outgoing packets on the interface connected to user terminals. The commandoutput shows that a large number of packets have been discarded, and the numberkeeps growing.
5.        Configure port mirroring tocopy incoming packets on the interface connected to the multicast server to anobserving port and use Wireshark to analyze the traffic. Determine whetherpacket loss has occurred due to traffic bursts.
Use Wireshark to analyze the incomingpackets on GE0/0/1 of Switch_1, finding that traffic bursts frequently occur.
Generally, the traffic rate is about 10Mbit/s but reaches about 1 Gbit/s at a specific time. Switch_1 needs to copythe traffic to different VLANs, so the downstream outgoing traffic rate exceeds1 Gbit/s bandwidth instantly. As a result, packet loss due to congestion.
If there is only a small amount of bursttraffic, increase the link bandwidth between network devices.
          The burst traffic rate hasreached 1 Gbit/s. As shown in Figure 3-1, the multicastserver hibernates for more than 1 second, and then sends video stream at a rateof about 1 Gbit/s. Several milliseconds later, it enters into hibernationagain. Although the average outgoing traffic rate is about 10 Mbit/s, thetraffic rate approximates to 1 Gbit/s if measured in milliseconds.
Figure 3-1 Video traffic burst on the multicast server

 
In this case, adjust the multicastpacket transmission mode on the multicast server to ensure that packets aresent at more even rates. Then check whether traffic bursts are mitigated, asshown in Figure3-2.
If the switch is running V200R001 ora later version, run the qos burst-modeenhanced command on related interfaces to increase the buffer capacity onthe interfaces. Then check whether packets are transmitted at even rates, withlittle traffic burst, as shown in Figure 3-2.
Figure 3-2 Traffic bursts migrated

 
Solution
l  Use cards providing higher rateto expand the capacity of the switches.
l  Configure an Eth-Trunk to bindto multiple physical ports to increase the port capacity.
l  Adjust the multicast source toensure that it sends multicast data packets more evenly, reducing trafficbursts.
3.13 Pixelization Occurs When Users Watch IPTV During Peak Hours

ApplicableProducts and Versions
S series switches of all versions
NetworkingDescription
In the following figure, the Switch isconnected to a multicast server, has GE1/0/0 added to Eth-Trunk 0, and isconnected to users. The multicast service is deployed on the Switch so thatusers can play videos.

FaultSymptom
Pixelization occurs when users watch IPTVduring peak hours.
CauseAnalysis
The VLAN priority in multicast packets is 5and changed to 4 on the inbound port, while rate limiting is configured on theinbound port for packets with the VLAN priority 4.
During peak hours, the actual traffic rateexceeds the configured traffic rate. As a result, some packets are discardedand pixelization occurs in videos.
TroubleshootingProcedure
1.        Check the configuration file,finding that priority mapping is configured on GE1/0/0 to map the priority 5 ofpackets to priority 4.
#
diffserv domain ds  //Create a DiffServdomain.
 8021p-inbound 4 phb ef green  //Map the 802.1p priority 4 of incoming VLANpackets to the PHB EF and mark these packets green.
 8021p-inbound5 phb af4 green  //Map the 802.1ppriority 5 of incoming VLAN packets to the PHB AF4 and mark these packetsgreen.
 8021p-outboundaf4 green map 5  //Map the PHB AF4 ofoutgoing VLAN packets marked green to the 802.1p priority 5.
 8021p-outbound af4 yellow map 5  //Map the PHB AF4 of outgoing VLAN packetsmarked yellow to the 802.1p priority 5.
 8021p-outbound af4 red map 5  //Map the PHB AF4 of outgoing VLAN packetsmarked red to the 802.1p priority 5.
 8021p-outbound ef green map 4  //Map the PHB EF of outgoing VLAN packetsmarked green to the 802.1p priority 4.
#
interface GigabitEthernet1/0/0
 eth-trunk 0
 trustupstream ds  //Apply the DiffServdomain to the interface.
#
2.        Check the configuration file,finding that rate limiting is configured on Eth-Trunk 0 for packets with thepriority 4.
#
traffic policy sp-wrr  //Create a trafficpolicy sp-wrr.
 classifier sp-wrr behavior sp-wrr  //Associate the traffic policy with thetraffic classifier sp-wrr andtraffic behavior sp-wrr.
traffic behavior sp-wrr  //Create atraffic behavior sp-wrr.
 carcir 102400 pir 102400 cbs 19251200 pbs 32051200 mode color-blind green passyellow pass red discard  //Configuretraffic policing: Set the CIR value to 102400 kbit/s, PIR value to 102400kbit/s, and PBS value to 32051200 bytes, permit green packets to be sent,permit yellow packets to pass through, and discard red packets.
traffic classifier sp-wrr operator and precedence 20  //Create a traffic classifier sp-wrr.
 if-matchvlan-8021p 4  //Match the packetswith the 802.1p priority 4.
#
interface Eth-Trunk0
 port link-type trunk
 port trunk allow-pass vlan 2 to 35003900 to 4094
 traffic-policysp-wrr inbound  //Apply the trafficpolicy sp-wrr in the inbounddirection of the Eth-Trunk.
 traffic-policy remark_cos outbound
#
3.        Modify the rate limitingconfiguration on Eth-Trunk 0.
4.        Modify or delete the trust upstream ds configuration onGE1/0/0.
Conclusion
A switch drops multicast data packets on aninterface or sends two copies of the same multicast flow to users, causingpixelation or artifacts on receiver hosts. Possible causes are as follows:
l  Rate limiting is configured onthe switch, and multicast data packets are dropped due to rate limit exceeding.
l  The instant packet rate exceedsthe interface bandwidth due to traffic bursts. As a result, multicast datapackets are dropped.
l  The rates of uplink anddownlink interfaces are different. For example, the uplink interface is anEth-Trunk consisting of multiple GE interfaces while the downlink interface isa GE interface, or the uplink interface is a 10GE interface, while the downlinkinterface is a GE interface.
l  The switch replicates packetsof the same group repeatedly and sends duplicate copies to downstream hosts.
Perform the following steps to locate andrectify the fault:
1.        Check whether rate limiting isconfigured on the uplink and downlink interfaces of the switch.
2.        Traffic bursts usually last fora short time, whereas the traffic rate on an interface is the average rate ofpackets calculated in a certain period. Therefore, it cannot be determinedwhether a traffic burst has occurred just according to the number of discardedpackets. You can obtain packets on the interface and use the Wireshark softwareto analyze obtained packet information. You can also ask R&D engineers tocheck chip registers on the switch so as to determine whether traffic burstshave occurred.
3.        Check whether the rates ofuplink and downlink interfaces match.
4.        Check whether Layer 3 multicastis configured on the switch. If so, the switch sends duplicate copies ofpackets to downstream hosts.
3.14 Image Pauses Occur When Users Watch Videos

ApplicableProducts and Versions
S2300&S3300S2700&S3700V100R005/V100R006
NetworkingDescription
In the following figure, Switch_1 isconnected to multiple fixed switches. Both Switch_1 and these fixed switchesbelong to VLAN 204 and have the multicast service enabled.

FaultSymptom
Image pauses frequently occur when users requestvideos. After users are directly connected to Switch_1, users can watch thevideos normally.
CauseAnalysis
Image pauses occur on receiver hosts,indicating that multicast data is not forwarded normally. Packet loss may haveoccurred or excessive multicast packets or duplicate multicast packets may beforwarded to the hosts.
This fault occurred when multicast packetswere lost due to heavy broadcast traffic.
TroubleshootingProcedure
1.        According to the informationreturned by the customer, determine that the fixed switches are faulty. Checkthe traffic on the inbound ports of the fixed switches, finding that thetraffic volume is high, there are excessive broadcast packets, and thebandwidth usage is more than 30%.
2.        There are also discarded packetstatistics on the fixed switches. Therefore, determine that the heavy broadcasttraffic results in multicast packet loss, affecting video quality.
Solution
Configure port isolation on Switch_1.
Conclusion
If many access switches are connected to anaggregation switch and belong to the same VLAN, you are advised to configureport isolation on the aggregation switch to prevent broadcast traffic fromoccupying bandwidth.


4 Packet Loss Prevention Measures


In the preceding sections, you can locatethe cause of packet loss on S series switches. To reduce packet loss, it isrecommended that you take the measures in Table 4-1during network deployment.
Table 4-1 Packet loss prevention measurements

   Cause
   
   Measure
   
  The PC connected to the remote device is  infected by virus.
  
  Scan virus on the PCs or servers  connected to the switch periodically.
  
  The hardware is faulty, for example,  physical link is faulty, card is loose, card is faulty, and interface status  is abnormal.
  
  l   Check whether the physical  link and cards are properly installed.
  l   The cable connected to the  electrical interface must be a standard cable. A non-standard cable may cause  link problems.
  l   Electrical interfaces must  work in full duplex mode. Half duplex mode will cause many problems.
  l   The optical module used on  the switch must be a Huawei-certified optical module.
  The optical modules from  different vendors use different implementation methods, and may cause  unexpected problems, such as packet loss. Therefore, only the  Huawei-certified optical modules can be used on Huawei switches.
  
  Burst traffic volume exceeds interface  bandwidth, causing congestion.
  
  l   Optimize packet transmission  on the server to reduce traffic burst.
  l   Configure an Eth-Trunk and  add member interfaces to the Eth-Trunk to load balance traffic and reduce  traffic on a single interface.
  
  Improper configuration
  
  /
  
  Network loop
  
  Plan the network configurations,  configure loop prevent protocol, and enable loop detection to prevent loops.
  l   Run the loopback-detect untagged mac-address ffff-ffff-ffff command in  the system view to broadcast BPDUs for loop detection and prevent them from  being terminated by unexpected devices.
  l   Run the loopback-detect enable command in the interface view to enable  loop detection.
  When the total number of VLANs  on the interfaces with loopback detection enabled exceeds 1024 VLANs, run the  loopback-detect action shutdown  command on these interfaces to set the action for a detected loopback to  shutdown. (The VLAN counter increases by 1 every time an interface is added to  a VLAN, even when multiple interfaces are added to the same VLAN.)
  
  Attack
  
  l   Configure ARP security to  protect the device against ARP or ARP Miss attacks.
  l   On the network prone to DHCP  and ARP attacks, such as campus networks, configure local attack defense  policies for DHCP and ARP protocol packets.
  
  The rate of packets sent to the CPU  exceeds the limit.
  
  The switch provides CPCAR values for each  protocol. Generally, the default CPCAR values can meet requirements. If  service traffic volume is too high, contact Huawei switch agentsHuawei  technical support engineers to adjust the CPCAR values.
  
  The product specifications are limited  and network planning is improper.
  
  When inter-card traffic volume is too  high, choose S9700S9300 and SRUB/SRUD as MPUs, to obtain higher inter-card  bandwidth.
  



5 Appendix


5.1  Configuring Traffic Statistics Collection
5.1 Configuring Traffic Statistics Collection

Overview
In Figure5-1, theS series switch supports interface-based and global statistics collection inthe inbound and outbound directions. The switch uses a traffic policy tocollect traffic statistics. When a traffic policy is applied to an interface,the switch collects statistics only about incoming or outgoing traffic on thisinterface; when a traffic policy is applied globally, the switch collectsstatistics about incoming or outgoing traffic on all interfaces.
The traffic policy configured in theinterface view takes precedence over the traffic policy configured in thesystem view. When traffic matches the traffic policy on an interface, thetraffic cannot match the global traffic policy. Therefore, traffic statisticsare not displayed.
Figure 5-1 Traffic statistics collection

 
Check the following information based ontraffic statistics:
1.        Whether traffic arrives at theinbound interface of the switch and whether packets are lost on the upstreamdevice.
2.        Whether traffic is forwarded tothe outbound interface of the switch and whether packets are lost on theswitch.
3.        Whether the Layer 2 and Layer 3information of traffic on the inbound interface of the switch is correct andwhether the upstream device has correctly encapsulated and forwarded packets.
4.        Whether the Layer 2 and Layer 3information of traffic on the outbound interface of the switch is correct andwhether the switch has correctly encapsulated and forwarded packets.
5.        Whether traffic is unstablebecause of MAC address flapping, route change, or IP address conflict.
ConfigurationProcedure
Configure ACL rules to match trafficstatistics.
Configure a traffic classifier to classifythe traffic matching ACL rules.
Configure the traffic behavior andconfigure traffic statistics collection in the behavior.
Configure the traffic policy, bind thetraffic classifier and behavior to the traffic policy, and apply the trafficpolicy to the inbound direction of the switch to implement traffic statisticscollection.
<HUAWEI> system-view
[HUAWEI] acl 3999
[HUAWEI-acl-adv-3999] rule permit icmp source 1.1.1.1 0 destination 2.2.2.20  //Configure ACL.
[HUAWEI-acl-adv-3999] quit
[HUAWEI] traffic classifier test operator or precedence 45  //Create a traffic classifier and referenceACL 3999 to the traffic classifier.
[HUAWEI-classifier-test] if-match acl 3999 //
[HUAWEI-classifier-test] quit
[HUAWEI] traffic behavior test  //Createthe traffic behavior and enable traffic statistics collection.
[HUAWEI-behavior-test] statistic enable
[HUAWEI-behavior-test] quit
[HUAWEI] traffic policy test  //Create atraffic policy and bind the traffic classifier and behavior to the trafficpolicy.
[HUAWEI-classifier-test] classifier test behavior test
[HUAWEI-classifier-test] quit
[HUAWEI] interface GigabitEthernet1/0/1
[HUAWEI-GigabitEthernet1/0/1] traffic-policy test inbound  //Apply the traffic policy to the inbounddirection of the interface, or run the traffic-policy test global inboundcommand in the system view to apply the traffic policy globally.
# View the overall traffic statistics.
<Quidway> display traffic policy statistics interface GigabitEthernet 1/0/1inbound
 
Interface: GigabitEthernet1/0/1 
 Traffic policy inbound: test 
 Rule number: 1 
 Current status: OK! 
--------------------------------------------------------------------- 
 Board : 1   
Item       Packets                      Bytes  
--------------------------------------------------------------------- 
Matched      0                           0  
  +--Passed    0                          0  
  +--Dropped  0                           0  
    +--Filter   0                           0  
    +--CAR   0                           0  
# View traffic statistics based on rules.
<Quidway> display traffic policy statistics interface GigabitEthernet 1/0/1inbound verbose rule-base
 
Interface: GigabitEthernet1/0/1 
 Traffic policy inbound: test 
 Rule number: 1 
 Current status: OK! 
---------------------------------------------------------------------  
 Classifier: test operator or  
 Behavior: test 
 Board : 1
 rule 5 permit icmp source 192.168.1.3 0  
 Passed Packet  0,Passed Bytes  0 
 Dropped Packet  0,Dropped Bytes  0
After completing traffic statisticscollection, disable this function to reduce CPU load.
If the S3700 does not support trafficstatistics collection on the outbound interface, perform the following steps totroubleshoot a packet loss fault.
l  Collect traffic statistics onthe inbound interface of the remote device and compare the traffic statisticswith those on the inbound interface of the S3700 to check whether packetsare lost on the S3700.
l  Configure packet redirection onthe inbound interface to redirect packets to the outbound interface, and checkwhether packet loss occurs. If packet loss is occurring, theS3700forwards packets normally. If packet loss does not occur, packets are droppedby the S3700.
 

  • x
  • convention:

gululu
Admin Created May 8, 2017 09:44:12 Helpful(1) Helpful(1)

thanks!
  • x
  • convention:

Come on!
walidnawar
Created May 8, 2017 15:57:39 Helpful(0) Helpful(0)

Amazing
  • x
  • convention:

Try to Add Value anywhere anytime , Tomorrow never waits
Khalefa
Created Aug 6, 2017 10:46:03 Helpful(0) Helpful(0)

Thanks
  • x
  • convention:

Reply

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login