Hello everyone,
Today I’m going to introduce you the troubleshooting thoughts on Eth-Trunk unbalanced load balancing.
1. Check whether the number of Eth-Trunk member interfaces is an exponential multiple of 2.
Procedure
When the number of Eth-Trunk member interfaces is 2 to the power of exponent, the hash algorithm is the most even. If the number of Eth-Trunk member interfaces is not 2 raised to the power of the exponent, the hash result may be slightly uneven. You are advised to adjust the number of Eth-Trunk member interfaces.
<HUAWEI> display interface eth-trunk 10
Eth-Trunk10 current state : DOWN (ifindex: 1041)
Line protocol current state : DOWN
Description: CC--Kazahtelecom-msk
Switch Port, PVID : 1, TPID : 8100(Hex),
......
----------------------------------------------------------
PortName Status Weight
----------------------------------------------------------
10GE1/5/0/5 DOWN 1
10GE2/5/0/6 DOWN 1
----------------------------------------------------------
The Number of Ports in Trunk : 2
The Number of Up Ports in Trunk : 0
Note:
By default, per-flow hash is used on CE switches. If load balancing among Eth-Trunk member interfaces is uneven, you can adjust the local route selection mode in the outbound direction and the remote device determines the remote device's route selection mode in the inbound direction. In this case, you need to check and modify the remote device's route selection mode. In addition, when the load of Eth-Trunk members is unbalanced, consider the following scenarios:
a. Whether member interfaces reside on different devices (stack virtualization scenario)
b. Check whether ECMP hashing is performed before Eth-Trunk hashing.
c. Whether Eth-Trunk hash is configured on two levels devices.
For different scenarios, in some cases, the hash factor, hash algorithm, and hash offset universal-id need to be adjusted at the same time. In most cases, only one factor needs to be adjusted.
2. Check whether Eth-Trunk member interfaces reside on different devices.
By default, local preferential forwarding is enabled on an Eth-Trunk in a stack of CE switches. Local preferential forwarding indicates that traffic from the local chassis is preferentially forwarded through Eth-Trunk member interfaces by default. Local preferential forwarding can reduce the forwarding delay and stack link usage.
When local preferential forwarding is enabled, load balancing among Eth-Trunk member interfaces on the same chassis is even, but load balancing among Eth-Trunk member interfaces on different chassis is uneven. This is a normal phenomenon and no action is required.
If the customer requires load balancing among member interfaces of different devices, perform the following operations:
Procedure
a. Run the display interface eth-trunk command in any view to check whether the Eth-Trunk member interfaces reside on different devices.
b. If the Eth-Trunk member interfaces reside on different devices, disable the local preferential forwarding function first. To disable the local preference function, run the local-preference disable command in the Eth-Trunk interface view.
<HUAWEI> system-view
[~HUAWEI] interface eth-trunk 10
[*HUAWEI-Eth-Trunk10] local-preference disable
[*HUAWEI-Eth-Trunk10] commit
Note: After the local preferential forwarding function is disabled, some traffic is forwarded through the stack link between switches. Ensure that the link bandwidth is sufficient.
3. Adjust the load balancing factors of the Eth-Trunk.
Procedure
Check whether the characteristics of packets forwarded by the Eth-Trunk match the configured load balancing mode. If they do not match, for example, if the MAC address of the packets to be forwarded changes and the load balancing mode is set to src-ip, load balancing cannot be performed.
Perform the following steps to check and modify the configuration:
Check the packet characteristics.
Determine the packet change factor.
Check whether the MAC address, IP address, or label of the packet changes.
Determine the packet type.
Determining that the packet is an IP packet, an MPLS packet, or a Layer 2 packet.
Determine the packet forwarding mode.
Known unicast and unknown unicast packets are forwarded in different processes, and the default load balancing algorithms are different. Therefore, you need to check whether the packets to be forwarded are known unicast packets or unknown unicast packets.
Note: If the destination MAC address of the packet is not in the MAC address table, the packet is an unknown unicast packet.
Check the load balancing mode of the Eth-Trunk interface.
Check the load balancing mode of the Eth-Trunk interface based on the packet forwarding mode (known unicast or unknown unicast).
Check the load balancing mode of known unicast traffic.
Run the display eth-trunk trunk-id command to check the Hash arithmetic field or run the display this command in the Eth-Trunk interface view to check the configured load balancing mode.
<HUAWEI> display eth-trunk 1
Eth-Trunk1's state information is:
Working Mode: Normal Hash Arithmetic: dst-ip
Least Active-linknumber: 1 Max Bandwidth-affected-linknumber: 16
Operating Status: down Number of Up Ports in Trunk: 0
--------------------------------------------------------------------------------
PortName Status Weight
……
Note: Users can run the load-balance { dst-ip | dst-mac | random | round-robin | src-ip | src-mac | src-dst-ip | src-dst-mac | enhanced [ resilient ] profile profile-name } command in the Eth-Trunk interface view to configure the load balancing mode. The load balancing modes supported by different models are different.
Enhanced profile profile-name indicates that load balancing is performed on an Eth-Trunk based on a load balancing profile. After an enhanced load balancing profile is applied, the outbound interface is selected based on the load balancing mode specified in the profile. If the load balancing mode is ENHANCED, check the load balancing mode of the enhanced profile. The enhanced load balancing mode has only one global profile and takes effect for both known and unknown unicast packets. Different fields are selected for hash calculation based on packet types.
<HUAWEI> display load-balance profile
Load-balance Profile: default // The default value is the default enhanced profile name.
Packet HashField
---------------------------------------------------
IP src-ip dst-ip
l4-src-port l4-dst-port
IPv6 src-ip dst-ip
l4-src-port l4-dst-port
L2 src-mac dst-mac
MPLS top-label 2nd-label
Eth-Trunk universal-id(1) hash-mode(1)
Stack universal-id(1) hash-mode(1)
---------------------------------------------------
Interface List:
---------------------------------------------------
Eth-Trunk 1
Check whether the characteristics of forwarded packets match the load balancing mode.
If the packet characteristics do not match the current load balancing mode, modify the load balancing mode based on the traffic model on the live network. The more frequently this parameter changes in traffic, the more balanced the traffic in this load balancing mode is.
For example, when only one PC is used for the test, the source IP address and source MAC address remain unchanged, and only the destination IP address and destination MAC address change. In this case, if the load balancing mode is set to source IP address-based or source MAC address-based, traffic cannot be load balanced. So you need to change the load balancing mode to destination IP address-based or destination MAC address-based.
If the packet is a known unicast packet, run the following command to modify it:
<HUAWEI> system-view
[~HUAWEI] interface eth-trunk 1
[*HUAWEI-Eth-Trunk1] load-balance dst-mac
[*HUAWEI-Eth-Trunk1] commit
If the packets are unknown unicast packets, run the following command to modify the configuration:
<HUAWEI> system-view
[~HUAWEI] unknown-unicast load-balance mac
[*HUAWEI] commit
If the enhanced load balancing mode is used, check the packet type (such as IPv4, IPv6, MPLS, or Layer 2) and modify the load balancing mode of the corresponding packet type in the enhanced profile.
For example, modify the load balancing mode of IPv4 packets to dst-ip in the enhanced profile test.
<HUAWEI> system-view
[~HUAWEI] load-balance profile test
[*HUAWEI-load-balance-profile-test] ip dst-ip
[*HUAWEI-load-balance-profile-test] commit
4. Check whether ECMP and Eth-Trunk are both configured.
If the preceding adjustment does not take effect, check whether ECMP and Eth-Trunk are both configured. That is, the Eth-Trunk is a member interface of ECMP.
Procedure
a. Run the display ip routing-table command in any view, check the equal-cost route and its member interfaces in the routing table, check whether the Eth-Trunk is a member interface of an equal-cost route.
<HUAWEI> display ip routing-table
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : _public_
Destinations : 717 Routes : 7204
Destination/Mask Proto Pre Cost Flags NextHop Interface
192.168.10.0/24 Static 60 0 D 218.90.246.10 Eth-Trunk1
192.168.9.0/24 Static 60 0 D 218.90.246.118 Eth-Trunk2
192.0.0.0/8 Static 60 0 D 222.190.158.162 Eth-Trunk3
192.109.148.0/23 Static 60 0 D 218.90.246.122 Eth-Trunk4
192.168.10.3/32 Static 60 0 D 218.90.246.10 Vlanif1118
b. If both ECMP and Eth-Trunk are configured, check whether ECMP local preferential forwarding is enabled. By default, ECMP local preferential forwarding is disabled. If ECMP local preferential forwarding is configured, you can disable it. Procedure:
CE6870EI and CE12800 series switches: Run the undo ecmp local-preference enable command in the load-balance-profile view.
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[*HUAWEI-load-balance-profile-default] undo ecmp local-preference enable
[*HUAWEI-load-balance-profile-default] commit
CE8800&7800&6800&5800 series switches (except the CE6870EI): Run the undo local-preference enable command in the ECMP view.
<HUAWEI> system-view
[~HUAWEI] load-balance ecmp
[~HUAWEI-ecmp] undo local-preference enable
[*HUAWEI-ecmp] commit
c. Adjust load balancing factors. For details, see Configuration Guide > IP Unicast Routing > IP Routing Table Management > Configuring the ECMP Load Balancing Mode in the product documentation.
CE12800 and CE6870EI: Configure ECMP load balancing based on the source IP address and protocol number of IPv4 packets.
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[~HUAWEI-load-balance-profile-default] ip src-ip protocol
[*HUAWEI-load-balance-profile-default] commit
CE8800&7800&6800&5800 series switches (except the CE6870EI): Configure ECMP load balancing based on the source IP address and protocol number of IPv4 packets.
<HUAWEI> system-view
[~HUAWEI] load-balance ecmp
[~HUAWEI-ecmp] ipv4 src-ip protocol
[*HUAWEI-ecmp] commit
d. If the traffic is received from multiple interfaces, add the source interface of the packet as the hash factor.
For CE6870EI and CE12800 series switches:
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[~HUAWEI-load-balance-profile-default] eth-trunk src-interface
[*HUAWEI-load-balance-profile-default] commit
For CE8800&7800&6800&5800 series switches excluding the CE6870EI:
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[~HUAWEI-load-balance-profile-default] l2 src-interface
[*HUAWEI-load-balance-profile-default] commit
Note: Only TRILL packets (ingress node) and non-IPv4, non-IPv6, and non-MPLS Layer 2 packets are supported. If load balancing is still uneven after the preceding adjustment, adjust the universal-id value of the Eth-Trunk or ECMP.
Configuration method: Run the eth-trunk universal-id universal-id command in the load balancing profile view. On the CE6870EI and CE12800 series switches, the value of universal-id ranges from 1 to 16, and the default value is 11. On other models, the value of universal-id ranges from 1 to 8, and the default value is 1.
5. Check whether two-level load balancing exists.
As shown in Figure 1, traffic is load balanced on SwitchA, some traffic is load balanced to SwitchB, and SwitchB continues to perform load balancing, this scenario is called two-level load balancing. If the load balancing factors of SwitchA and SwitchB are the same, all traffic forwarded from SwitchB to SwitchC is balanced on one member interface, causing uneven load balancing. This phenomenon is called hash polarization.
Figure 1 Two-level load balancing
For example, if SwitchA receives eight flows with different characteristics (1 to 8), the following hash result is displayed in the outbound direction: Traffic 1 and traffic 5 with the same characteristics are distributed to link 1, traffic 2 and traffic 6 with the same characteristics are distributed to link 2, and so on. SwitchB and SwitchA use the same hash algorithm. Therefore, when SwitchB receives traffic with signatures 1 and 5, it distributes the traffic to link a instead of link a and link b. This results in hash polarization.
To solve the hash polarization problem in two-level load balancing scenarios, prevent devices at both ends from using the same load balancing parameters.
Procedure
a. If the traffic has multiple characteristics, the two levels of devices can use different hash factors. For example, the first-level Eth-Trunk uses the source IP address for hash, and the second-level Eth-Trunk uses the destination IP address for hash.
The first-level Eth-Trunk interface uses the source IP address for hash calculation.
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[~HUAWEI-load-balance-profile-default] ip src-ip
[*HUAWEI-load-balance-profile-default] commit
The second-level Eth-Trunk uses the destination IP address for hash calculation.
<HUAWEI> system-view
[~HUAWEI] load-balance profile default
[~HUAWEI-load-balance-profile-default] ip dst-ip
[*HUAWEI-load-balance-profile-default] commit
b. If the effect is not obvious after the hash factor is adjusted, you can adjust the two-level hash algorithm to different algorithms.
Configuration method: Run the eth-trunk hash-mode hash-mode-id command in the load-balance-profile view. The hash-mode-id value ranges from 1 to 13 on the CE6870EI and CE12800 series switches, from 1 to 5 on the CE5810EI, and from 1 to 9 on other models. The default value is 1.
c. If the problem persists, adjust the value of universal-id.
Configuration method: Run the eth-trunk universal-id universal-id command in the load balancing profile view. On the CE6870EI and CE12800 series switches, the value of universal-id ranges from 1 to 16, and the default value is 11. On other models, the value of universal-id ranges from 1 to 8, and the default value is 1.
This is what I want to share with you today, thank you!
