[Problem Description]
During NDMP backup prefetch, NAS resources may be exhausted when a large directory is backed up. As a result, the controller is reset for self-healing.
[Symptom Description]
During NDMP backup prefetch, NAS resources may be exhausted when a large directory is backed up. As a result, the controller is reset for self-healing, and NAS services are interrupted.
[Cause]
The NDMP backup prefetch operation READDIR_PLUS applies for a large number of memory resources and delivers a prefetch task to the SPACE module for execution. The SPACE layer generates multiple prefetch tasks. The number of concurrent prefetch tasks reaches the maximum. As a result, the prefetch operation responds slowly. Resources applied for by upper-layer NDMP cannot be released in time and are exhausted. As a result, the controller resets and services are interrupted.
[Location Method]
1. Check whether the NDMP backup service is configured on the storage device.
2. Search for the keyword READDIR_PLUS in the message log at the time when the problem occurs. If the corresponding READDIR_PLUS TIMEOUT record exists, the READDIR_PLUS execution times out.
[2018-06-11 18:34][210756.503014] [][15000028a0000][WARN][(NAS_FSA)[367 suppressed] Nas_vfs:(READDIR_PLUS)TIMEOUT.][PROTO_OMAGENT][nas_vfs_timer_timeout,139][swapper/16]
[2018-06-11 18:34][210756.596578] [][15000028a0000][WARN][(NAS_FSA)[372 suppressed] Nas_vfs:opcode(READDIR_PLUS)TIMEOUT(9).][PROTO_OMAGENT][nas_vfs_timer_del,211][CSD_11]
3. Search for the keyword NAS_MEM. The function that allocates the most memory is displayed as count: xxxxx,func:ostorvfs_readdir_plus_netlink.
[Solution]
1. Emergency method: Disable NDMP pre-reading (NDMP pre-reading cannot be disabled completely).
developer:/>change service ndmp_config readdir_plus_enable=no
1) For full backup, this switch can be used to disable the prefetch function in the backup process, while the prefetch function in the reporting process remains enabled.
2) For incremental backup, the prefetch function of the marking process can be disabled, and the prefetch function of the reporting process can be enabled.
Description:
Marking process: When an incremental backup is performed, the system marks whether the file is modified based on the time of the file.
Reporting process: What files are backed up to the backup software?
Backup process: Backing up files to tape
2. Solution:
V300R006C20SPH002 can be installed in V300R006C20.
[Post-Recovery Check]
NA
[Appendix]
NA
[Applicability]
OceanStor V3 series
