Symptom
Some IVS services on the CSP portal do not exist. As a result, the system displays a message indicating that the backup fails.
Impact on Services
The automatic backup of the same service every day fails.
Cause
After capacity expansion rollback or decoupling deployment rollback, the CSP automatically backs up the rolled back container, causing the backup failure.
1. During container registration, the IVS sends the nodeid injected by the CSP to the container environment variable as the unique identifier of the container to the CSP. The CSP uses the nodeid in the backup table to indicate the container instance to be backed up. A new value is generated when the container is recreated and restarted.
2. When the decoupling deployment rollback or capacity expansion rollback (not involved in 8.1) is successful, the CSP destroys the container instances on related servers, but does not clear the container instance information in the backup task table. As a result, the overall backup task fails.
Identification Method
1. On the CSP portal, choose Application Lifecycle Deployment, click the IVS application, and click the Execution Logs tab.
2. If rollback is performed for capacity expansion or decoupling deployment and the name of the IVS service that fails to be backed up is displayed on the backup page, copy the service name and choose Resource Management > Edge Site Details > Containers. In this case, the problem occurs, for details, see Emergency Maintenance.
Note: Ensure that the backup is complete. Otherwise, the services to be backed up may be deleted. As a result, normal services will not be backed up.
3. If the service exists on the Resources > Edge Site Details > Containers page, a backup failure message is displayed. Contact the IVS region director.
Emergency Operations
Step 1 Log in to the CSP OM portal, choose Application Configuration > Services, find the GaussDB service whose status is M, and copy the value of podName.
Step 2 Choose Resource Management > Edge Site Management > Containers and find the node where the pod is located based on the value of podName.
Step 3 Use an SSH tool to log in to the external IP address of the node in step 2 as user cspexpert and switch to user root.
Step 4 Run the following command to access the backup service database:
cid=$(docker ps | grep cspgauss | grep -v grep | awk -F "" '{print $1}') ;docker exec -it -u root ${cid} su - dbuser -c "gsql -d backupdb -U backupdbuser -p 22080 -W Cspdbg@2017"
Step 5 Run the following SQL statement to manually delete the services that fail to be backed up:
Note: Replace autobackup20191015142911 with the name of the automatic backup task.
delete TBL_BACKUPOBJ where SERVICENAME in (select b.SERVICENAMEfrom TBL_BACKUP_TASK a inner join TBL_BACKUPOBJ b ona.SERVICENAME=b.SERVICENAME and BACKUPSTATUS='Failed 'and TASKNAME='autobackup20191015142911');
step 6 After executed successfully, the system displays a message indicating the number of rows of data deleted. The number is the same as the number of records that fail to be backed up in step 5.
Manually start a backup task to check whether all backups are successful.
