Repeated Upgrade and Rollback Failures

Problem Symptom

After the upgrade package is uploaded and the upgrade is successful, roll back the software. After the system software is upgraded and rolled back repeatedly for multiple times, the upgrade is suspended or paused.

Problem Diagnosis

Check the /var/log/deploy/console/run/servicetool_console.log run log of the console. The log file shows that error java.lang.OutOfMemoryError: unable to create new native thread is printed.

Causes

After each upgrade and rollback, residual threads exist in the DeployManager process. After multiple upgrade and rollback operations, the number of accumulated threads reaches the upper limit of the operating system or the memory is insufficient. As a result, new threads fail to be created during an upgrade.

Solution

Run the following commands to restart the DeployManager process:

tomcatpid=`ps -ef | grep tomcat | grep "/opt/fusionstorage/deploymanager/jre/bin/java" | awk '{print $2}'` ;

kill -9 ${tomcatpid} ;

bash /opt/fusionstorage/deploymanager/Tomcat/WebTomcat/bin/tomcat_cron.sh;