Symptom:
In the CIFS and NFS services with a load greater than 600 MB for a long time, the CPU usage can reach over 90%, which does not reach the disk performance bottleneck, and the performance cannot be further improved. In extreme cases, the system may be suspended.
Analysis:
The coredump information is as follows:
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.051975] <IRQ> [<ffffffffa15760e0>]? tcp_pkt_to_tuple+0x0/0x70 [nf_conntrack]
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.051988] [<ffffffff8139d030>]? kretprobe_trampoline+0x0/0x50
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.051992] [<ffffffff81072fdb>]? tick_sched_timer+0x5b/0xb0
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.051997] [<ffffffff8106761a>]? __run_hrtimer+0x4a/0x100
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052000] [<ffffffff810678f7>]? hrtimer_interrupt+0xc7/0x200
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052004] [<ffffffff810367c7>]? wake_affine+0x357/0x3b0
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052009] [<ffffffff8101b217>]? smp_apic_timer_interrupt+0x67/0xa0
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052014] [<ffffffffa159d03e>]? nf_nat_get_offset+0x5e/0x90 [nf_nat]
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052020] [<ffffffff81003a93>]? apic_timer_interrupt+0x13/0x20
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052027] [<ffffffffa15760e0>]? tcp_pkt_to_tuple+0x0/0x70 [nf_conntrack]
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052032] [<ffffffffa1567ffd>]? ipt_do_table+0x1bd/0x61c [ip_tables]
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052035] [<ffffffffa156818e>]? ipt_do_table+0x34e/0x61c [ip_tables]
2015 Apr 29 12:16:26 NH123_02 [warning] kernel: [320879.052039] [<ffffffff8139b819>]? _spin_lock_bh+0x9/0x30
According to the analysis of the stack that is continuously printed on the CPU in the message, the upper-layer service process frequently invokes the software interrupt of the network port when sending messages. However, when the service pressure is heavy, the service process execution takes a long time. As a result, the service process execution is slow. It is always in the software interrupt processing of the network port driver or packet filtering of the network port (firewall). Therefore, it is suspected that there are too many firewall rules on the NAS. As a result, the packet filtering cost on the firewall is too high. According to the statistics collected by the perf tool, the CPU usage of firewalls is over 30%, and the CPU usage of some cores is over 40%.
The firewall rules are further analyzed. For connections established on the, the firewall directly accepts the rules.
Rule A: -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
As services are enabled or disabled, firewall rules are added or deleted. (For example, NDMP, FTP, SMIS, and HTTP are newly deployed.) A maximum of 1000 firewall rules may exist. New rules can only be placed on the top. Rule A may be moved to the bottom after running for a long time. As a result, firewall rules need to be traversed for each packet processing, consuming a large number of CPU resources.
Solution:
When the firewall rule changes, delete rule A and add it again to ensure that rule A is always in the head of the linked list.
Summary:
This problem applies not only to NAS but also to client performance tuning with a large number of firewall rules.
