[Insider Sharing] Troubleshooting high CPU load on iStack S6700 switch series Highlighted

Latest reply: Nov 12, 2019 04:34:38 2915 2 0 1

Recently I troubleshoot a case on TAC related to high CPU load. We had to identify the root-cause for the high load and take some measures to lower down the CPU load value. The device affected was a S6700, you can find the version details down below:

[Insider Sharing] Troubleshooting high CPU load on iStack S6700 switch series-1059671-1

Usually when you spot high CPU events for your system “display cpu-usage” command output it will help you to identify the CPU task that is presenting high values.

Let’s ***yze below image,  CPU usage is quite high, 67%, but this value is not impacting data traffic in order for the users to feel the network/services are performing bad. Huawei switches use a dual-plane structure. In the structure, the control plane and forwarding plane are separated from each other, and data forwarding is completed by the forwarding engine on the forwarding plane. In this situation, a high CPU usage on the control plane does not affect data forwarding on the forwarding plane. If the high CPU usage on the control plane leads to protocol flapping, the forwarding table is modified, and data forwarding is affected.

Secondly the task usage shows XSTP task occupying a part of CPU resource.

 

VIDL is the system IDLE task and it’s normal to have a high value.

This type of task is a special task in the system. It is named VIDL, has the lowest priority, and occupies the CPU only when all the other tasks are idle. When a non-idle task needs to occupy the CPU, the VIDL task cannot occupy the CPU. CPU usage of the VIDL task indicates a lower system CPU usage and idler system.

 

OS task is the operating system task and it’s normal to see it holding 30% of CPU.

 

XSTP is holding 17% and this is abnormal. This task is associated to spanning tree calculation. We will have to look into spanning tree configuration in order to find the root-cause.

[Insider Sharing] Troubleshooting high CPU load on iStack S6700 switch series-1059671-2

Browsing through configuration we can see that the mode of working is VBST. This is the only spanning tree related configuration we can find into the system configuration.

#

stp mode vbst

#

 

We will have to dig deeper. Another useful command for troubleshooting high CPU events issues is “display cpu-defend statistics”. This will show if the system receives too many packets belonging to a specific protocol type. In our situation  we can see that the system is not dropping any VBST related packets, so we can conclude that the system is not receiving too many VBST packets in a very short time. We will need to look somewhere else.

[Insider Sharing] Troubleshooting high CPU load on iStack S6700 switch series-1059671-3

Checking the Spanning Tree process in detail I manage to identify that the number of PVs is quite high. By using “display stp brief” command I manage to count more than 660 entries. The PV quantity is the number of VBST-enabled interfaces multiplied by the number of VLANs. If the PV quantity exceeds the specifications, the CPU usage may exceed the threshold. As a result, the switch cannot process tasks in a timely manner, protocol calculation is affected. The specification for S6700 is about 300 PV, but if you are using an iStack system the specification increase based on the number of items added to stack. For example in our case we have 2 units in the stack so specification is about 600 PVs.

 

To solve the problem the VLAN assignment was optimized so 600 PVs where finally deployed and kept the CPU load at normal values. Unused VLAN were remove from trunk interfaces.

 [Insider Sharing] Troubleshooting high CPU load on iStack S6700 switch series-1059671-4

 

I hope you enjoy this case!

  • x
  • convention:

user_2790689
Created Apr 23, 2016 01:40:06 Helpful(0) Helpful(0)

Thank you
  • x
  • convention:

marco2287
Created 7 days ago Helpful(0) Helpful(0)

How must be calculated the PV value?

If you have S5720HI switch of 48 port and 40 vlan setup in the VRP with VBST enable on all inerfaces. The value is 48x40=1920? or just is calculate based on the trunk interfaces and include all the Eth-Trunks interfaces?
  • x
  • convention:

Telecomunication Engineer.  CCNA Cisco Instructor
Certified.

Comment

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login