Failed to Change the FSM Management IP Address

Hi team, here's a new case.

Problem Symptom

When you run a DSwareTool command to modify the management IP address of an FSM node, or modify the management IP address of an FSM node and that of an FSA node concurrently, The system has an internal error. is displayed in the command output.

Problem Diagnosis

N/A

Causes

The IP address of the OMM component fails to be changed.
The configuration file of the FSA node fails to be modified.
The configuration file and management IP address of the FSM node fail to be modified.
The management IP address of the object service fails to be modified (applicable only to the object service).

Solution

Modify the FSM node configuration.

Log in to an FSM node.
If the FSM node is deployed on a VM, log in to the FSM node using VNC. If the FSM node is deployed on a physical server, log in to the FSM node using the BMC of the server as user dsware.

The default passwords of users dsware and root are IaaS@OS-CLOUD9! and Huawei@123, respectively.
Run the following command and enter the password of user root to switch to user root:
su - root

Run the following command to obtain parameters required for changing the management IP address:

grep "/cluster/managerip/modify" /var/log/oam/fsm/manager/run/log-fsm-audit.log

[2015-09-09 11:10,044] [Request] [user=admin] [ip=192.168.40.253] [url=/cluster/managerip/modify; method=POST; msg={"netPlaneType":"external","gateway":"192.168.40.1","netMask":"255.255.254.0","om_active_ip":"192.168.40.15","om_standby_ip":"192.168.40.16","om_float_ip":"192.168.40.14","hosts":[{"old_net_addr":"192.168.40.11","new_net_addr":"192.168.40.21"},{"old_net_addr":"192.168.40.12","new_net_addr":"192.168.40.22"}]}][2015-09-09 11:10,417] [Response] [url=/cluster/managerip/modify; resultCode=200; msg={"result":1,"detail":[{"ip":"192.168.40.12","errorCode":32501020,"description":"Failed to change the management IP address."}]}]

Information similar to the preceding is displayed. Table Description describes the meaning of the parameters in Request displayed in the command output.

Response shows the IP address of the failed node and provides details about the failure.

../Resource/public_sys-resources/note_en.png?v=20200527

If the command output contains multiple IP address change records, obtain the proper one based on the change time.

**Table1** Description
No.	Parameter Value	Description
Parameter 1	netPlaneType=external	Network plane type of the IP address to be changed. The value can be internal or external. external indicates that the IP address of the external network plane is to be changed, and internal indicates that the IP address of the internal network plane is to be changed.
Parameter 2	om_active_ip = 192.168.40.15	New IP address of the active FSM node.
Parameter 3	om_standby_ip = 192.168.40.16	New IP address of the standby FSM node.
Parameter 4	om_float_ip = 192.168.40.14	New FSM floating IP address.
Parameter 5	gateway = 192.168.40.1	New FSM gateway.
Parameter 6	netMask = 255.255.254.0	New FSM subnet mask.
Parameter 7	hosts=[{old_net_addr = 192.168.40.11, new_net_addr = 192.168.40.21}, {old_net_addr = 192.168.40.12, new_net_addr = 192.168.40.22}]	Old and new IP addresses of the FSA node.

After running each shell script in the following steps, you need to run the echo $? command. This checks whether the exit code of a script is 0.

If yes, go to the next step.
If no, contact engineers to locate the fault.

Run the following command on the current FSM node to check whether it is the active node:

bash /opt/dfv/oam/oam-u/ha/ha/module/hacom/script/get_harole.sh

Check whether active is displayed in the command output.

If yes, go to 8.
If no, go to 6.

If the FSM node is deployed on a VM, log in to the FSM node using VNC. If the FSM node is deployed on a physical server, log in to the FSM node using the BMC of the server as user dsware.

Run the following command and enter the password of user root to switch to user root:

su - root

Run the following command to switch to user omm:

su - omm -s /bin/bash

The database password is different each time the database is installed. Go to the /opt/dfv/oam/oam-u/oam-gaussdb/bin/ directory, run the sh gaussdb_change_pwd.sh omm command as user root to change the password of user omm, and use the new password to log in to the database.

Run the following command to switch to the database CLI:

gsql -U omm -d omm -p 21600

Enter the password for logging in to the database.

If you enter incorrect passwords for three consecutive times, the database will lock the omm account for about 270 seconds.

Run the following commands to query the IP addresses in the FSM data table:

set search_path=dsware_1;

select * from tbl_server_info;

Use the same method to check whether the management IP addresses in the tbl_service_server_media_info, tbl_insulated_server, and tbl_obs_server_info tables have been updated. If no data exists in the tables or the management IP addresses have been updated, you do not need to run the update commands in 12 (applicable to the object service and file service).

Check whether the IP addresses in the FSM data table are new.

If yes, go to 13.
If no, go to 12.

Run the following commands to change the IP addresses in the FSM database:

update tbl_server_info set MANAGEIP = 'New management IP address' where MANAGEIP = 'Old management IP address';

Example: update tbl_server_info set MANAGEIP = '192.168.40.20' where MANAGEIP = '192.168.40.10';

update tbl_service_server_media_info set NODEMANAGEIP= 'New management IP address' where NODEMANAGEIP= 'Old management IP address';

Example: update tbl_service_server_media_info set NODEMANAGEIP= '192.168.40.20' where NODEMANAGEIP= '192.168.40.10';

update tbl_insulated_server set MANAGERIP = 'New management IP address' where MANAGEIP = 'Old management IP address';

Example: update tbl_insulated_server set MANAGEIP = '192.168.40.20' where MANAGEIP = '192.168.40.10';

update tbl_obs_server_info set MANAGEMENTINTERNALIP= 'New management IP address' where MANAGEMENTINTERNALIP= 'Old management IP address';

Example: update tbl_obs_server_info set MANAGEMENTINTERNALIP= '192.168.40.20' where MANAGEMENTINTERNALIP= '192.168.40.10';

If no file service node exists in the environment, you do not need to update the tbl_service_server_media_info and tbl_insulated_server tables.

If no object service node exists in the environment, you do not need to update the tbl_obs_server_info table.

Enter \q to exit the database CLI.

Run the following command to exit the omm user interface:

exit

Change the management IP address of the object service. (This step is applicable only to the file service. For the file service and block service, skip this step.)

user_name indicates super administrator admin, and password indicates the password of user admin.

Run the following command to change the management IP address of the object service:
curl -i -k -H "Content-type: application/json" -H "Accept: application/json" -H "user-agent: ibase-client" -H "X-Auth-Token: NDAwNDYwNTM3M0FGRkZGRkZGRkZGRkZG" -X PUT https://Floating IP address of the management plane:8088/dfv/service/oam/mdfNodeOamIp -d '{"oamFloatIp":"New floating IP address","masterOamIp":"New management IP address of the first FSM node","slaveOamIp":"New management IP address of the second FSM node","oamIpList":"{\"Old management IP address\":\"New management IP address\"}"}' -k
X-Auth-Token is x_auth_token obtained in the previous step.
If you do not need to change the values of the related parameters, enter the original values.
If FSM is deployed with a single plane, delete slaveOamIp from the command.
If you want to change the management IP addresses of multiple nodes, you can add multiple IP address pairs separated by commas (,) to oamIpList. The IP address pairs are as follows: {\"Old management IP address 1\":\"New management IP address 1\",\"Old management IP address 2\":\"New management IP address 2\",···}.

Run curl -i -k -H "Content-type: application/json" -H "Accept: application/json" -H "user-agent: ibase-client" -X POST -d '{"user_name":"admin", "password":"xxx"}' https://Floating IP address of the management plane:6066/api/v2/aa/sessions to obtain x_auth_token.

Perform the required operation based on network plane type:

To query the network plane type, run the following command on the FSM node: cat /opt/dsware/manager/webapps/dsware/WEB-INF/manager-ha.properties | grep "double_plane". If double_plane=false is returned in the command output, FSM adopts a single-plane deployment. Otherwise, a dual-plane deployment is adopted.

Check whether the FSM node is deployed with a single plane.

If the management IP address to be changed is the IP address of the internal management plane:
If the management IP address to be changed is the IP address of the external management plane:

If yes, run the following command: bash /opt/dsware/manager/script/modifyMgntIpInCfg.sh New floating IP address New management IP address of the first FSM node New management IP address of the second FSM node New subnet mask New gateway false
If no, run the following command: bash /opt/dsware/manager/script/modifyMgntIpInCfg.sh New management IP address of the FSM node New management IP address of the FSM node -- New subnet mask New gateway false
If you do not need to change the values of the related parameters, enter the original values.

If yes, run the following commands on the active and standby nodes:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New floating IP address New management IP address of the first FSM node New management IP address of the second FSM node New gateway New subnet mask internal (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsa_modify_ip.sh New floating IP address New management IP address of the first FSM node New management IP address of the second FSM node
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh
If no, run the following commands:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New management IP address of the FSM node New management IP address of the FSM node -- New gateway New subnet mask internal (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsa_modify_ip.sh New management IP address of the FSM node New management IP address of the FSM node --
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh

If yes, run the bash /opt/dsware/manager/script/restartLocalHa.sh double modifyIp command.
If no, run the bash /opt/dsware/manager/script/restartLocalHa.sh single modifyIp command.

If FSM is deployed with a single plane, perform the following operations:
If FSM is deployed with dual planes, perform the following operations:

Go to the /opt/dfv/oam/public/fsm/modifyIp directory and manually generate the modified_ip.ini file. If the file exists, delete it first. Write the management IP addresses of the FSM and FSA nodes to be modified. First, write a row containing a pair of old and new FSA management IP addresses, and then a row containing a pair of old and new internal FSM management IP addresses. The format is as follows:
```
old_fsa_ip_1=new_fsa_ip_1old_fsm_ip_2=new_fsm_ip_2...
```
Check whether the management node is deployed in active/standby mode and run the specific command to modify the FSM configuration file.
If yes, run the following command: bash /opt/dsware/manager/script/modifyInnerIpInCfg.sh New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node New internal subnet mask New internal gateway false
If no, run the following command: bash /opt/dsware/manager/script/modifyInnerIpInCfg.sh New internal management IP address of the FSM node New internal management IP address of the FSM node -- New internal subnet mask New internal gateway false
Check whether the management node is deployed in active/standby mode and run the specific commands to modify the OMM configuration file.
If yes, run the following commands on the active and standby nodes:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node New internal gateway New internal subnet mask internal (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsa_modify_ip.sh New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh
If no, run the following commands:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New internal management IP address of the FSM node New internal management IP address of the FSM node --New internal gateway New internal subnet mask internal (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsa_modify_ip.sh New internal management IP address of the FSM node New internal management IP address of the FSM node --
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh
If the management node is deployed in active/standby mode, run the following command on the primary node to synchronize file data:
/opt/dfv/oam/oam-u/ha/ha/module/hacom/tools/ha_client_tool --syncallfile
Check whether the management node is deployed in active/standby mode and run the specific command to restart the HA process:
If yes, run the bash /opt/dsware/manager/script/restartLocalHa.sh double modifyIp command.
If no, run the bash /opt/dsware/manager/script/restartLocalHa.sh single modifyIp command.
Check whether the management node is deployed in active/standby mode and run the specific command to modify the FSM configuration file.
If yes, run the following command: bash /opt/dsware/manager/script/modifyMgntIpInCfg.sh New external floating IP address New external management IP address of the first FSM node New external management IP address of the second FSM node New external subnet mask New external gateway false
If no, run the following command: bash /opt/dsware/manager/script/modifyMgntIpInCfg.sh New external management IP address of the FSM node New external management IP address of the FSM node -- New external subnet mask New external gateway false
Check whether the management node is deployed in active/standby mode and run the specific commands to modify the OMM configuration file.
If yes, run the following commands on the active and standby nodes:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New external floating IP address New external management IP address of the first FSM node New external management IP address of the second FSM node New external gateway New external subnet mask external (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh
If no, run the following commands:
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_fsm_modify_ip.sh New external management IP address of the FSM node New external management IP address of the FSM node --New external gateway New external subnet mask external (If an IPv6 management network is used, enter a prefix to replace the subnet mask.)
bash /opt/dfv/oam/public/fsm/modifyIp/oam_u_iptables_config.sh
If the management node is deployed in active/standby mode, run the following command on the primary node to synchronize file data:
/opt/dfv/oam/oam-u/ha/ha/module/hacom/tools/ha_client_tool --syncallfile
Check whether the management node is deployed in active/standby mode and run the specific command to restart the HA process:
If yes, run the bash /opt/dsware/manager/script/restartLocalHa.sh double modifyIp command.If no, run the bash /opt/dsware/manager/script/restartLocalHa.sh single modifyIp command.
Go to the /opt/dfv/oam/public/fsm/modifyIp directory and manually generate the modified_ip.ini file. If the file exists, delete it first. Write the management IP addresses of the FSM and FSA nodes to be modified. First, write a row containing a pair of old and new FSA management IP addresses, and then a row containing a pair of old and new FSM management IP addresses. The format is as follows:
```
old_fsa_ip_1=new_fsa_ip_1old_fsm_ip_2=new_fsm_ip_2...
```
Check whether the management node is deployed in active/standby mode and run the specific command to modify the FSM configuration file.

1. If FSM is deployed in active/standby mode, run the following command to check whether the current FSM node is the first or second FSM node:
cat /opt/omm/oms/workspace/webapps/dsware/WEB-INF/manager-ha.properties | grep local_name |awk -F "=" '{print $2}'
In the command output, if ha1 is displayed, the current node is the first FSM node. Otherwise, the node is the second FSM node.
2. In the VM fault recovery scenario, you do not need to perform any operation on the standby node or synchronize files.
Check whether the management node is deployed in active/standby mode and run the specific commands to modify the OMM configuration file.
If the management node is deployed in active/standby mode, run the following command on the active node to synchronize file data.
/opt/dfv/oam/oam-u/ha/ha/module/hacom/tools/ha_client_tool --syncallfile
Check whether the management node is deployed in active/standby mode and run the specific command to restart the HA process:

Modifying the FSA node configuration (applicable to the block service)

Modify the FSA node configuration file if the following conditions are met:

Modify the internal management plane when the FSM node is deployed in single-plane or dual-plane mode.
Modify the floating IP address or management IP addresses of FSM nodes.

Obtain the FSA modification result from the command output in 3.
The IP address in detail of the Response message indicates the node that fails to be modified.
Check whether any FSA node whose IP address fails to be changed exists.

If yes, go to 3.
If no, no further action is required.

Use PuTTY to log in to the first FSA node whose IP address fails to be changed through its old management IP address, and then switch to user root.

Check whether the management node is deployed in active/standby mode and run the specific command to modify the OMM configuration file.

If the related parameters are not modified, retain the original values.

If yes, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node Management IP address of the FSA node
If no, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal management IP address of the FSM node New internal management IP address of the FSM node -- Management IP address of the FSA node

Check whether the floating IP address is changed.

If yes, go to 6.
If no, go to 13.

Check whether dsware_manager_ip in the /opt/fusionstorage/agent/conf/dsware_manager_ip file is the new floating IP address of FSM nodes.

If the floating IP address is not the new one, run the following command to change the floating IP address:

sed -i "s/dsware_manager_ip=.*/dsware_manager_ip=New IP address/" /opt/fusionstorage/agent/conf/dsware_manager_ip

Check whether the old floating IP address exists in the /opt/fusionstorage/agent/conf/dsware_cluster_info file.

If the old floating IP address exists, run the following command to change the floating IP address:

sed -i "s/Old floating IP address/New floating IP address/" /opt/fusionstorage/agent/conf/dsware_cluster_info

Check whether the old floating IP address exists in the /opt/dsware/vbs/conf/vbs_cluster_info file.

If the old floating IP address exists, run the following command to change the floating IP address:

sed -i "s/Old floating IP address/New floating IP address/" /opt/dsware/vbs/conf/vbs_cluster_info

Check whether the vbs_Old floating IP address_conf.cfg configuration file exists in the /opt/dsware/vbs/conf/ directory.

If yes, run the following command to rename the file and then restart the dsware_vbs process (run the ps command to search for the process and kill it):

mv /opt/dsware/vbs/conf/vbs_Old floating IP address_conf.cfg /opt/dsware/vbs/conf/vbs_New floating IP address_conf.cfg

Note: If the configuration file is not named in the format of vbs_Old floating IP address_conf.cfg, skip this step.

Run the bash /opt/fusionstorage/agent/script/dswareAgent_tool.sh restart command to restart the FSA process.

Check whether the volume_attach_inf configuration file exists in the /opt/fusionstorage/agent/conf/ directory.

If yes, go to 12.
If no, go to 13.

Check whether the old floating IP address of FSM nodes exists in the volume_attach_inf file.

If the old floating IP address exists, run the following command to change the floating IP address:

sed -i "s/Old floating IP address/New floating IP address/" /opt/fusionstorage/agent/conf/volume_attach_inf

After the modification, restart the operating system of the node. After the operating system is restarted, run the lsscsi command to check whether the volume is successfully mounted.

If yes, go to 13.
If no, contact technical support engineers.

Check whether node_manager_ip in the /etc/node_manager_ip file is the new management IP address of the FSA node.

If the management IP address is not the new one, run the following command to change the management IP address:

sed -i "s/node_manager_ip=.*/node_manager_ip=New IP address/" /etc/node_manager_ip

Run the following command to restart the FSA and OMA processes:

bash /opt/fusionstorage/agent/script/dswareAgent_tool.sh restart

If the management IP address of the physical server where the FSA node is deployed is not changed, restarting the process will fail. Change the management IP address of the physical server in a timely manner. After the change, the FSA and OMA processes automatically start.

Repeat 1 to 14 to modify the configuration of other FSA nodes whose IP addresses fail to be changed.

Modifying the configuration of an FSA node (applicable to the OceanStor 100D 8.0.3 object service)

Modify the FSA node configuration file if the following conditions are met:

Modify the internal management plane when the FSM node is deployed in single-plane or dual-plane mode.
Modify the floating IP address or management IP addresses of FSM nodes.

Obtain the FSA modification result from the command output in 3.
The IP address in detail of the Response message indicates the node that fails to be modified.
Check whether any FSA node whose IP address fails to be changed exists.

If yes, go to 3.
If no, no further action is required.

Use PuTTY to log in to the first FSA node whose IP address fails to be changed through its old management IP address, and then switch to user root.

Check whether the management node is deployed in active/standby mode and run the specific command to modify the OMM configuration file.

If the related parameters are not modified, retain the original values.

If yes, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node Management IP address of the FSA node
If no, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal management IP address of the FSM node New internal management IP address of the FSM node -- Management IP address of the FSA node

Check whether the floating IP address is changed.

If yes, go to 6.
If no, go to 7.

Check whether dsware_manager_ip in the /opt/fusionstorage/agent/conf/dsware_manager_ip file is the new floating IP address of FSM nodes.

If the floating IP address is not the new one, run the following command to change the floating IP address:

sed -i "s/dsware_manager_ip=.*/dsware_manager_ip=New IP address/" /opt/fusionstorage/agent/conf/dsware_manager_ip

Run the bash /opt/fusionstorage/agent/script/dswareAgent_tool.sh restart command to restart the FSA process.

Check whether node_manager_ip in the /etc/node_manager_ip file is the new management IP address of the FSA node.

If the management IP address is not the new one, run the following command to change the management IP address:

sed -i "s/node_manager_ip=.*/node_manager_ip=New IP address/" /etc/node_manager_ip

Run the following command to instruct ZooKeeper to change the management IP address:

bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh update_zk_conf update_zk_cluster_info New FSA management IP address

Run the following command to restart the FSA and OMA processes:

bash /opt/fusionstorage/agent/script/dswareAgent_tool.sh restart

If the management IP address of the physical server where the FSA node is deployed is not changed, restarting the process will fail. Change the management IP address of the physical server in a timely manner and restart the NIC. After the restart, the FSA and OMA processes automatically start.

Repeat 1 to 10 to modify the configuration of other FSA nodes whose IP addresses fail to be changed.

Modifying the configuration of an FSA node (applicable to OceanStor 100D 8.0.3 file service)

Modify the FSA node configuration file if the following conditions are met:

Modify the internal management plane when the FSM node is deployed in single-plane or dual-plane mode.
Modify the floating IP address or management IP addresses of FSM nodes.

Obtain the FSA modification result from the command output in 3.
The IP address in detail of the Response message indicates the node that fails to be modified.
Check whether any FSA node whose IP address fails to be changed exists.

If yes, go to 3.
If no, no further action is required.

Use PuTTY to log in to the first FSA node whose IP address fails to be changed through its old management IP address, and then switch to user root.

Check whether the management node is deployed in active/standby mode and run the specific command to modify the OMM configuration file.

If the related parameters are not modified, retain the original values.

If yes, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal floating IP address New internal management IP address of the first FSM node New internal management IP address of the second FSM node Management IP address of the FSA node
If no, run the following command: bash /opt/fusionstorage/agent/script/dsware_agent_handle.sh oma_operate config_oma_ip New internal management IP address of the FSM node New internal management IP address of the FSM node -- Management IP address of the FSA node

Check whether the floating IP address is changed.

If yes, go to 6.
If no, go to 9.