Hello everyone!
Today, I'd like to share a case.
Problem Description
All the clients were reported to disconnect from U2000 sever. Also server ping found ok from the network.
Problem Analysis
1- Check the client and server connectivity found ok.
2- Check the U2000 server processes and found database is stopped.
3- After running the database visibility of the network restored.
Root Cause
The physical memory of the system was full which causes the data base to stop working.
The physical memory is not enough Ha-monitor need to monitor the server status, but we didn’t get the status successful , it report time out when we run this command svc_adm –cmd status –f because this is the part of the HA system, it trigged the switch from current server to standby server.
The HA monitor killed the process of U2000. Why we query status failed? It seems related the physical memory usage, the system run out of the physical memory , the usage running in high level at 98% for long time. In this server , the memory size is only 32G.
Here is the logs from server From the Ueasy Heath check ,we find that this U2000 manage acess and trans devices ,but as you feedback ,the U2000 only have trans device .so please confirm if the U2000 manage trans devices, if not, the U2000 server have already reached the limit of the management , from the below table, we can see when manage multi-domain case ,only 8000 equivalent NES can be managed .
In fact ,the U2000 have already managed 7454 Equivalent NEs Number, at the same time ,there are some unnecessary process running . Analysis from the data we collected, the root cause is the physical memory size is not enough, it is only 32GB. Once NMS started , the physical memory usage reached 80%-90%.
Once we did some collect task, it will over 90%.
This U2000 have already managed 7454 Equivalent NEs Number, reached the max capacity of 8000.
Solution Description
1- U2000 server reached the Max capacity with ne management, need to upgrade the physical memory to void this issue happen again from 32G to 64G as per NMS requirement.
2- Didn’t add more other devices in this U2000 when it already reaches near threshold value in this case 8000 devices.
3- Optimize U2000 server processes and stop and remove all unnecessary/unwanted server processes to release the memory of the system.
4- Another way to work around this issue, is to complete the HA system, but this is not a solution we can only use in emergency only as workaround.
Welcome to leave a message below.
We’ll study together.
Thank you!
