[Insider Sharing] eSight fails to backup S5700 switch configuration

Hello everybody,

Today I comeback with a new Insider Sharing thread, spoiler :) It's about eSight.

When doing network maintenance, backup the configuration file from network elements, it's a basic requirement and it's usually performed regularely. This operation can be done in hundreds of ways, manually and automatically.

When you have 100+ network elements in your garden, definitely you want to do it automatically, unless you have loads of free time.

A nice and easy approach is to setup the NMS to perform a backup configuration task every week at a specific time, and store the configuration on the server. I will not spend too much time explaining how to configure this feature, I will just send a link from eSight documentation which explains how to perform this operation. See below:

http://support.huawei.com/hedex/pages/EDOC1000158488JEG07078/08/EDOC1000158488JEG07078/08/resources/cfgexample/n_cfgexample_502.html?ft=0&fe=10&hib=6.1.12.10.4.1&id=n_cfgexample_502&text=Example%20for%20Configuring%20eSight%20to%20Periodically%20Back%20Up%20Device%20Configuration%20Files&docid=EDOC1000158488

I would like to gather your attention with a troubleshooting case. It's refering exactly to backup configuration task, for one of the elements from the network it failed to be executed.

Software version:

eSight V300R005C00SPC303

Software switch: VRP5.16 V200R008C00SPC500 - stack of 3 S5700-28X-PWR-LI-AC

Symptom:

eSight returns the following error message

[Insider Sharing] eSight fails to backup S5700 switch configuration-2698571-1

Troubleshooting steps:

1. Check the SNMP and SSH configuration on the network element, it's seems to be correct, here there is a snip. SSH it's required because the transfer between eSight and S5700 switch is performed by SFTP. Also related port SSH, SNMP ad FTP must be opened if you have a firewall in the middle.

snmp-agent

snmp-agent local-engineid 800007DB0330D17E72A0D5

snmp-agent community write cipher %^%#.`i{*@VNS+n~y5Yut$9W-M3b*k)ON8nL;=H0&jF#<"=PVJ8d#:8FyV)SS_c)]GZnDI-_T%SRzn~8\2i6%^%#

snmp-agent community read cipher %^%#~D`3*c~*(#\ZgmG;ts%5vGku;7Yx8/C"W""xf"$N[N264j@egY0</8I.2Yq%|'D{!WSsO3^5cf0r>*|C%^%# mib-view iso-view

snmp-agent sys-info contact xxxxx

snmp-agent sys-info location xxxxx

snmp-agent sys-info version v2c

undo snmp-agent sys-info version v3

snmp-agent target-host trap address udp-domain 10.1.11.10 params securityname cipher %^%#z;%PR!3kX-L,;w=~P:ISN!\Q<pTg$XCoumD1m]YW%^%#

snmp-agent target-host trap address udp-domain params securityname cipher %^%#MRQ~0sLb\S%8^mXlG)yCf}T83Y/>X&oDj/TmjNL-%^%# v2c

snmp-agent mib-view included iso-view iso

snmp-agent inform timeout 5

snmp-agent inform resend-times 6

snmp-agent inform pending 7

snmp-agent trap enable

ssh server compatible-ssh1x enable

stelnet server enable

ssh authentication-type default password

ssh user xxxxx

ssh user xxxxx authentication-type password

ssh user xxxxx service-type stelnet

ssh client first-time enable

2. Change the protocol from SFTP to FTP and after TFTP to rule out the transfer protocol. The same error message was displayed in the other 2 situations. This means I should focus my troubleshooting on SNMP protocol.

3. Debug SNMP protocol.

A very common sequence for debugging is the following

<R5_U28_S5700_ACC>debugging snmp-agent process

<R5_U28_S5700_ACC>debugging snmp-agent header

<R5_U28_S5700_ACC>debugging snmp-agent event

<R5_U28_S5700_ACC>debugging snmp-agent packet

<R5_U28_S5700_ACC> t d

<R5_U28_S5700_ACC> t m

Start the backup task. Log the screen to collect the output.

To close debug:

<R5_U28_S5700_ACC>u t d

Info: Current terminal debugging is off.

<R5_U28_S5700_ACC>undo deb al

Info: All possible debugging has been turned off.

This will allow to capture all SNMP related data during the backup task.

From debugging I was unable to see any SNMP SET request packets reaching the switch to trigger the SNMP backup. So I decide to do a packet capture on eSight side.

4. The packet capture reveal something interesting.

SNMP SET packets are properly generated by eSight. Let's look closer in the SNMP SET packet.

[Insider Sharing] eSight fails to backup S5700 switch configuration-2698571-2

Do you notice something wrong in the packet?

Yes! the community parameter string is black... This means that on eSight write community was not set and community write is not matching. Without write rights, the switch will never accept an SNMP SET packet from NMS side.

SNMP SET packets are critical for this process, because it will save first the current configuration and then it will send it to FTP server.

Solution

I was lucky to see that empty string for SNMP write community, I only need to reconfigure with the correct string to match the one from the switch and the backup was succesful.

Conclusion

For this type of cases, first thing to do is to check if community string is matching for both read and write. Then make sure the UDP/TCP ports for SSH, FTP, SNMP are open.

If there is NAT between NMS and NE networks, addresses and ports are rewritten, the TCP sequence number will be changed, and thereafter the NAT service needs to maintain this delta in SEQ and ACK numbers by performing sequence NAT on all subsequent packets. In this situation you have to enable NAT ALG on the NAT device in order to automatically allocate data ports and firewall permissions for dynamic data connection and to rewrite the control packets with the appropriate NAT address and port information.

I hope to enjoy reading this case! Bye!