Got it

What Do I Do If "READ is not supported in state standby" Is Displayed in the Log?

Latest reply: May 18, 2022 03:48:29 2691 2 2 0 0

Hello, everyone!

This post will introduce you to what means if "READ is not supported in state standby" is displayed in the log?

Applicable Versions

6.5.x

Symptom

The log of the HDFS server or client contains the following: Operation category READ is not supported in state standby.

2017-06-07 19:01:44,529 | WARN| [Thread-1] | Exception encountered while connecting to the server: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby | org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:711)


Fault Locating

When the client connects to the server for the first time, the client does not know which NameNode is the active node. The connection process is as follows:

1. When the HDFS client is connected to the server, the client reads from the hdfs-site.xml configuration file the dfs.nameservices.mappings information to obtain the instance number of the NameNode, for example, 13 and 14 in the following result.

<property>      <name>dfs.nameservices.mappings</name>      <value>[{"name":"hacluster","roleInstances":["14","13"]}]</value>    </property>

2. The client selects the NameNode with a smaller number, for example, NameNode 189-39-234-149 corresponding to dfs.namenode.https-address.hacluster.13.

<property>      <name>dfs.namenode.rpc-address.hacluster.13</name>      <value>189-39-234-149:25003</value>   </property>   <property>      <name>dfs.namenode.rpc-address.hacluster.14</name>      <value>189-39-235-85:25003</value>   </property>

3. If the NameNode 189-X-X-149 is active, the client continues to process service requests.

4. If the NameNode 189-X-X-149 is standby, the server returns the client a message stating that the server is in the standby state and records the following log:

2017-06-16 15:59:53,537 | INFO  | IPC Server handler 384 on 25000 | IPC Server handler 384 on 25000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 10.X.X.85:57486 Call#0 Retry#0:       org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby | Server.java:2102

In this case, the client prints the following information, indicating that it connects to a standby NameNode and therefore it attempts to connect to another NameNode.

17/06/16 16:03:25 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over 189-X-X-149/10.X.X.149:25000. Trying to fail over immediately.   org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

5. The client then attempts to connect to another NameNode with a larger instance ID (dfs.namenode.http-address.hacluster.14 in the example).

<property><name>dfs.namenode.rpc-address.hacluster.13</name><value>189-X-X-149:25003</value></property><property><name>dfs.namenode.rpc-address.hacluster.14</name><value>189-X-X-85:25002</value></property>

If NameNode 189-X-X-85 is active, the service request can be processed normally.

6. [In the NameNode active and standby scenario] If NameNode 189-X-X-85 is standby, the client prints the following information and attempts to reconnect to another NameNode.

17/06/16 17:40:47 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over 189-X-X-85/10.X.X.85:25000 after 1 fail over attempts. Trying to fail over after sleeping for 907ms.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
...
17/06/16 17:40:48 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over 189-X-X-149/10.X.X.149:25000 after 2 fail over attempts. Trying to fail over after sleeping for 1433ms.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

7. If the two NameNodes are in the standby state, the client repeatedly connects to the two NameNodes for 10 times. If one of the two NameNodes is restored to the active state at a connection in the 10 attempts, the connection is successful. Otherwise, the connection fails after 10 attempts.

17/06/16 17:42:09 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over 189-X-X-149/10.X.X.149:25000. Not retrying because failovers (10) exceeded maximum allowed (10)org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

8. The maximum number of times that the two NameNodes are reconnected and the increasing waiting time are specified by the following parameters in hdfs-site.xml in the client configuration file:

<property><name>dfs.client.failover.max.attempts</name><value>10</value></property><property><name>dfs.client.failover.sleep.base.millis</name><value>500</value></property>

Conclusion

The "Operation category READ is not supported in state standby" log is printed by the client when it connects to a standby NameNode. The client will automatically reconnect to NameNodes and therefore this log generally requires no special attention.

However, if this log is printed for a large number of times (for example, for 10 consecutive times in the following) and many failovers occur, the client fails to connect to NameNodes or the NameNode servers are in the dual-standby state. You need to, in this case, troubleshoot the network problems between the client and NameNode or abnormal status of the NameNode.

17/06/16 17:42:09 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over 189-X-X-149/10.X.X.149:25000. Not retrying because failovers (10) exceeded maximum allowed (10)   org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

That's all, thanks!


  • x
  • convention:

olive.zhao
Admin Created May 18, 2022 02:43:53

Thanks for your sharing!
View more
  • x
  • convention:

Suhail
Created May 18, 2022 03:48:29

Great work is done, thanks for sharing
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.