[Problem Description]
Trigger conditions:
1. The time of the NTP server on the network changes. For example, the time of the NTP server changes due to the week number rollover of the GPS. As a result, the NTP clients connected to the NTP server synchronize the new time of the NTP server.
2. The security devices are configured to synchronize time from the NTP server.
Fault symptom:
The time of the NTP server interconnected with the security devices changes. As a result, the time of the security devices changes.
Identification method:
1. NIP2000/5000 series
Log in to the web UI as an administrator, choose System > Configuration > Clock, and check whether the NTP service is enabled and the NTP server is configured. If the NTP service is enabled, the device may be affected. If the NTP service is disabled, the device will not be affected.
2. Other Security Gateway
Run the display current-configuration | include ntp command in the user view to check the NTP configuration on the device.
Step 1 If the NTP configuration does not exist, the problem described in this warning will not occur. If NTP is configured, go to step 2.
Step 2 If the device uses the local clock and the stratum is 1, the problem will not occur. Otherwise, go to step 3. The following configuration indicates that the device uses the local clock and the stratum is 1:
<HUAWEI>display current-configuration | include ntp ntp-service refclock-master 1 |
Step 3 If the NTP client is configured, the problem described in this warning will occur. If one or more of the following configurations exist, the NTP client is configured:
<HUAWEI>display current-configuration | include ntp ntp-service unicast-server ntp-service unicast-peer ntp-service manycast-client ntp-service broadcast-client ntp-service multicast-client |
[Root Cause]
The week number rollover of the GPS may cause the time rollover of the NTP server. If the function of synchronizing time with the NTP server is configured on the network, firewalls on the network synchronize time with the NTP server. As a result, the time of the firewalls will roll back.
[Impact and Risk]
1. Errors will occur during certificate verification for functions such as IPSec and SSL VPN certificate authentication.
2. The policy time range and blacklist aging functions will become invalid.
3. The timestamps of all logs change to the time after the rollover. As a result, log information will be disordered, affecting log query and O&M.
[Measures and Solutions]
1. Preventive measure:
Take the following preventive measures before April 6, 2019:
Measure 1: Locate the GPS device on the network, and check whether the WNRO risk exists. If the risk exists, take measures to prevent it.
Measure 2: If preventive measures cannot be taken on the GPS device or the GPS device cannot be found, identify the NTP server with the highest stratum on the network. Configure this NTP server to use the local clock source. For details about the operations on Huawei datacom devices, see preventive measure 2 in the precaution notice (ENW-P-C-2019054/CIP-P-201922).
For details about the operations on non-Huawei NTP servers, see the corresponding operation guide.
Measure 3: If the preceding two preventive measures cannot be taken, perform the following operations on each NTP client one by one. (This measure requires a lot of workload. Therefore, measure 1 or 2 is strongly recommended.)
1. NIP2000/5000 series
Log in to the web UI before April 6, 2019, disable NTP time synchronization, set Configuration Mode to Manually Set the Time, and set the device time to the current time.
Other Security Gateway
1) Run the display current-configuration | include refclock-master command in the user view. Record the configuration for subsequent restoration.
<HUAWEI>display current-configuration | include refclock-master ntp-service refclock-master 3 |
2) Run the ntp-service refclock-master 1 command in the system view to configure the device to use the local NTP clock source.
[HUAWEI]ntp-service refclock-master 1
3) Run the display ntp-service status command in the user view and check that clock stratum is 1 and reference clock ID is LOCAL.
<HUAWEI>display ntp-service status clock status: synchronized clock stratum: 1 reference clock ID: LOCAL(0) nominal frequency: 100.0000 Hz actual frequency: 100.0000 Hz clock precision: 2^17 clock offset: 0.0000 ms root delay: 0.00 ms root dispersion: 10.95 ms peer dispersion: 10.00 ms reference time: 13:27:02.829 UTC Mar 20 2019(E03CBE26.D43BF727) synchronization state: clock set |
2. Restoration of preventive measures:
After the GPS WNRO problem is solved, enable the NTP time synchronization function. Perform the following operations:
1. NIP2000/5000 series
Log in to the web UI, choose System > Configuration > Clock, and enable NTP time synchronization again.
2. Other Security Gateway
1) If there is not refclock-master configuration in step 1 of measure 3, then delete the refclock-master configuration that is added in the prevention solution.
[HUAWEI]undo ntp-service refclock-master
Otherwise run the ntp-service refclock-master xx command in the system view.
[HUAWEI]ntp-service refclock-master xx //xx is the value recorded in step 1 of measure 3.
2) Run the display ntp-service status command in the user view and ensure that clock status is synchronized and clock stratum is correct.
<HUAWEI>display ntp-service status clock status: synchronized clock stratum: 3 reference clock ID: LOCAL(1) nominal frequency: 100.0000 Hz actual frequency: 100.0000 Hz clock precision: 2^17 clock offset: 0.0000 ms root delay: 0.00 ms root dispersion: 10.97 ms peer dispersion: 10.00 ms reference time: 13:23:22.581 UTC Mar 20 2019(E03CBD4A.94DA5DAF) synchronization state: clock set but frequency not determined |
3. Emergency recovery measures:
If preventive measures are not taken properly and services are affected due to the GPS WNRO problem, take the following emergency measures:
Measure 1: Locate the GPS device that may cause the GPS WNRO problem on the network and remove the risk.
Measure 2: If preventive measures cannot be taken on the GPS device or the GPS device cannot be found, identify the NTP server with the highest stratum on the network. Configure this NTP server to use the local clock source and set the correct local time. For details about the operations on Huawei datacom devices, see the precaution notice (ENW-P-C-2019054/CIP-P-201922).
For details about the operations on non-Huawei NTP servers, see the corresponding operation guide.
Measure 3: If the preceding two preventive measures cannot be taken, perform the following operations on each NTP client one by one. (This measure requires a lot of workload. Therefore, measure 1 or 2 is strongly recommended.)
1. NIP2000/5000 series
Log in to the web UI as an administrator, choose System > Configuration > Clock, disable NTP time synchronization, use the local device clock, and set the device time to the current time. Save the configuration and restart the device to restore services.
2. Other Security Gateway
1) Run the display current-configuration | include refclock-master command in the user view. Record the configuration for subsequent restoration.
<HUAWEI>display current-configuration | include refclock-master ntp-service refclock-master 3 |
2) Run the ntp-service refclock-master 1 command in the system view to configure the device to use the local NTP clock source.
[HUAWEI]ntp-service refclock-master 1
3) Run the display ntp-service status command in the user view and check that clock stratum is 1 and reference clock ID is LOCAL.
<HUAWEI>display ntp-service status clock status: synchronized clock stratum: 1 reference clock ID: LOCAL(0) nominal frequency: 100.0000 Hz actual frequency: 100.0000 Hz clock precision: 2^17 clock offset: 0.0000 ms root delay: 0.00 ms root dispersion: 10.95 ms peer dispersion: 10.00 ms reference time: 13:27:02.829 UTC Mar 20 2019(E03CBE26.D43BF727) synchronization state: clock set |
4) Run the clock datetime HH:MM:SS YYYY-MM-DD command to change the time of the security devices to the correct time.
4. Restoration of emergency recovery measures:
After the GPS WNRO problem is solved, enable the NTP time synchronization function. Perform the following operations:
1. NIP2000/5000 series
Log in to the web UI, choose System > Configuration > Clock, and enable NTP time synchronization again.
2. Other Security Gateway
1) If there is not refclock-master configuration in step 1 of measure 3, then delete the refclock-master configuration that is added in the prevention solution.
[HUAWEI]undo ntp-service refclock-master
Otherwise run the ntp-service refclock-master xx command in the system view.
[HUAWEI]ntp-service refclock-master xx //xx is the value recorded in step 1 of measure 3.
2) Run the display ntp-service status command in the user view and ensure that clock status is synchronized and clock stratum is correct.
<HUAWEI>display ntp-service status clock status: synchronized clock stratum: 3 reference clock ID: LOCAL(1) nominal frequency: 100.0000 Hz actual frequency: 100.0000 Hz clock precision: 2^17 clock offset: 0.0000 ms root delay: 0.00 ms root dispersion: 10.97 ms peer dispersion: 10.00 ms reference time: 13:23:22.581 UTC Mar 20 2019(E03CBD4A.94DA5DAF) synchronization state: clock set but frequency not determined |
