[Insider Sharing] Ultimate network troubleshooting - remote port mirroring

4252 0 0 0

Hi Guys,


Sometimes, network problems require special troubleshooting methods. Display commands, debugging, traffic statistics may not be enough to closing the issue, so a capture with protocol packets passing through a port or matching a specific traffic flow, it can prove to be very helpful. But, what would you do if the target interface/flow is far away in your network and there is no way to travel fast enough to that point of presence and plug your monitoring station directly into device? Remote port mirroring will do the trick. 


For starters, what is packet mirroring? 


Is a feature that helps you to capture packets passing through an online interface.

How it work? 

It copies packets going through an interface and send them out to another. 

Why we need this?

-     - for troubleshooting purposes;

-     - for network appliances that require monitoring of traffic;

-     - for educational purposes. 

In remote port mirroring, interface that collects and sends the packets to the ***yzing host is not connected locally to the mirroring device. The monitoring device and the observing port are connected through a Layer 2 network.  

How it works in this case? Have you heard about RSPAN?

Remote switched port ***yzer (RSPAN): The device encapsulates packets passing through mirrored ports into VLAN packets, and the observing port broadcasts the VLAN packets in the RSPAN VLAN to forward packets to the monitoring device. It is mandatory to deploy RSPAN VLAN from mirrored port device to observing port device through your layer 2 network. 


How to do it on Huawei network? 

Let's consider a simple scenario, with some 27&57&77 series Huawei Switches. Topology:

             [Insider Sharing] Ultimate network troubleshooting - remote port mirroring-1307849-1


S7700's interface eth2/0/1 is configured with mirrored port role for inbound and outbound traffic. 


#

interface Ethernet2/0/1

port link-type access

port default vlan 58

port-mirroring to observe-port 3 inbound

port-mirroring to observe-port 3 outbound

#


On system-view is necessary to configure observing port and RSPAN vlan. S7700 switch encapsulates mirrored packets into VLAN packets, and then, observing port will broadcasts VLAN packets into the RSPAN VLAN and forward them  to monitoring device. 


#

observe-port 3 interface Ethernet2/0/4 vlan 999

#


It is mandatory to deploy RSPAN vlan over layer 2 network, from the mirrored to observing device, so I must add it to trunks.


#

interface Ethernet2/0/4

port hybrid tagged vlan 999

#



S5700, is an intermediary should and it should relay vlan 999 further to S2700.


#

interface GigabitEthernet0/0/2

port link-type trunk

port trunk allow-pass vlan 999

ntdp enable

ndp enable

#

interface GigabitEthernet0/0/3

port link-type trunk

port trunk allow-pass vlan 999

ntdp enable

ndp enable

#



On S2700, gi0/0/2 will collect copied packets from RSPAN vlan. Interface gi0/0/3 connects to monitoring host. It is configured to untag packets received on RSPAN vlan.


#

interface GigabitEthernet0/0/2

port link-type trunk

port trunk allow-pass vlan 999

ntdp enable

ndp enable

interface GigabitEthernet0/0/3

port link-type trunk

port trunk pvid vlan 999

port trunk allow-pass vlan 127 333

#


On the mirrored port, I have generated an uniform traffic consisting 6 packets outbound and 16 packets inbound.


<S7700>display interface Ethernet2/0/1

----------------------------------------------------------------------------------------------

Last 10 seconds input rate 194832 bits/sec, 16 packets/sec

Last 10 seconds output rate 75568 bits/sec, 6 packets/sec

----------------------------------------------------------------------------------------------


With this current configuration, I can see mirrored packets leaving S7700 interface and reach S5700 inbound

 

< S7700>display interface Ethernet2/0/4

-----------------------------------------------------------------------------------------

Last 10 seconds input rate 176 bits/sec, 0 packets/sec

Last 10 seconds output rate 261600 bits/sec, 22 packets/sec

-----------------------------------------------------------------------------------------


[S5700]display interface GigabitEthernet 0/0/2

--------------------------------------------------------------------------------------------------

Last 10 seconds input rate 261600 bits/sec, 22 packets/sec

Last 10 seconds output rate 0 bits/sec, 0 packets/sec

--------------------------------------------------------------------------------------------------


But I cannot see any packets going outbound observing interface. What could be the reason?

 

[S5700]display interface GigabitEthernet 0/0/3

----------------------------------------------------------------------------------------

Last 10 seconds input rate 0 bits/sec, 0 packets/sec

Last 10 seconds output rate 0 bits/sec, 0 packets/sec

-----------------------------------------------------------------------

Is clear that frames arrive on the switch. The next step is to check and match its destination MAC address value with an entry in switch MAC address table.

 

Let’s check mac-address table for vlan 999 to see how looks like:

 

<S5700>display mac-address dynamic vlan 999

-------------------------------------------------------------------------------

MAC Address    VLAN/VSI                          Learned-From        Type     

-------------------------------------------------------------------------------

aaaa-bb1e-5a54 999/-                               GE0/0/2             dynamic  

aaaa-bb11-cccc  999/-                               GE0/0/2             dynamic  

aaaa-bb22 5a54 999/-                               GE0/0/2             dynamic  

………………………………………………………………………………………………

 

So no entry learned from gigabitethernet0/0/3. It means that traffic will be returned back to GigabitEthernet0/0/2, and because destination mac is actually source mac frame will be dropped.

Because of the learning function, mac-addresses are populating the table. It records the source MAC address and inbound interface of the Ethernet frame in a MAC address entry. When switch is receiving other Ethernet frames destined for this MAC address, the S5700 forwards the frames through the corresponding outbound interface according to the MAC address entry.

When doing bidirectional mirroring, inbound and inbound packets will be copied and delivered through observed port.  Those packets will arrive on intermediary switch but here switch will drop ethernet frames which come through the interface which has learned these frame’s destination mac-address. Usually switch broadcast frames that don’t match any entry in the mac-table but because that destination is seen as source, packets will be dropped on gi0/0/3.

The solution here is to disable mac-address learning on RSPAN vlan. So no mac-address entry will be created in mac-address table so all traffic will be flooded on RSPAN vlan.

 

[S5700]vlan 999

[S5700-vlan999]mac-address learning disable

[S5700]undo mac-address dynamic vlan 999  \\ mac-adress need to be aged out instantly

 

Using this command sequence, mac address table will be empty and layer 2 forward will be made broadcasting all the frames in RSPAN vlan 999.

In this way packets will be send out through Gi0/0/3

 

[S5700]display interface GigabitEthernet 0/0/3
-----------------------------------------------------------------------------------------
Last 10 seconds input rate 0 bits/sec, 0 packets/sec
Last 10 seconds output rate 287216 bits/sec, 24 packets/sec

----------------------------------------------------------------------------------------

Conclusion:

For remote mirroring is necessary to disable mac-address learning on RSPAN VLAN on intermediary devices, otherwise packets will be returned back by intermediary switches and finally dropped. 

 

Hope to enjoy reading this case :)


  • x
  • convention:

Reply

Reply
You need to log in to reply to the post Login | Register

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!

Login and enjoy all the member benefits

Login
Fast reply Scroll to top