HDFS RPC Server Summary

Hello, everyone!

The post will share with you the HDFS RPC server.

1. RPC Server Type

The following table lists the RPC server types and related configuration parameters in HDFS. The \org\apache\hadoop\hdfs\server\namenode\NameNodeRpcServer.java file describes declaration and initial configuration of the RPC server.

2. clientRpcServer Parameter Description

2.1. dfs.namenode.rpc-bind-host

Specify the actual address bound to the RPC server. If this address is set, the host name part of dfs.namenode.rpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

2.2. dfs.namenode.rpc-address

Specify the RPC address for processing all client requests. If there are multiple NameNode processes for HA/Federation, you need to add the NameService ID to the parameter name, for example, dfs.namenode.rpc-address.ns1. The value is in the format of nn-host1:rpc-port.

2.3. dfs.namenode.handler.count

Listen to the number of NameNode RPC server threads of requests from clients. If dfs.namenode.servicerpc-address is not configured, the NameNode RPC server threads need to listen to requests from all nodes.

3. serviceRpcServer Parameter Description

3.1. dfs.namenode.servicerpc-bind-host

Specify the actual address bound to the service RPC server. If this parameter is set, the host name part of dfs.namenode.servicerpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

3.2. dfs.namenode.servicerpc-address

Specify the RPC address used for communication with the HDFS service. If the parameter is set, BackupNode, Datanodes, and all other services are connected to the address. If there are multiple NameService processes for HA/Federation, you need to add the NameService ID to the parameter name, for example, dfs.namenode.servicerpc-address.ns1. The value is in the format of nn-host1:rpc-port. If the parameter is not set, the value of dfs.namenode.rpc-address is used as the default value.

3.3. dfs.namenode.service.handler.count

Listen to the number of NameNode RPC server threads of requests from DataNode and other non-client nodes. dfs.namenode.service.handler.count is valid only when dfs.namenode.servicerpc-address is configured.

4. lifelineRpcServer Parameter Description

4.1. dfs.namenode.lifeline.rpc-bind-host

Specify the actual address bound to the lifeline RPC server. If this address is set, the host name part of dfs.namenode.lifeline.rpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

4.2. dfs.namenode.lifeline.rpc-address

Specify the NameNode lifeline RPC address. This is an optional RPC address, which can be used to perform health check and node status reporting, thus preventing resource exhaustion in the main RPC processing program pool. If there are multiple NameNode processes for HA/Federation, you need to add the NameSer ID to the parameter name, for example, dfs.namenode.lifeline.rpc-address.ns1. The value is in the format of nn-host1:rpc-port. If this parameter is not set, NameNode does not start the lifeline RPC server. By default, this parameter is not set.

4.3. dfs.namenode.lifeline.handler.ratio

Specify the ratio of dfs.namenode.lifeline.handler.count to dfs.namenode.handler.count. The value of dfs.namenode.handler.count is the function of the number of DataNode nodes in the cluster. This parameter is used to automatically adjust the number of threads processed by the lifeline RPC server. As the message processing is lightweight, the number of threads processed by the lifeline RPC server is less than that required by the active NameNode RPC server. If dfs.namenode.lifeline.handler.count is configured, this parameter is not used. dfs.namenode.lifeline.handler.count specifies the absolute number of threads. If dfs.namenode.lifeline.rpc-address is not configured, this parameter does not take effect.

4.4. dfs.namenode.lifeline.handler.count

Specify the absolute number of threads running on the NameNode lifeline RPC server. These threads are used to process the lifeline protocol of DataNode and HA health check requests from ZKFC. If this parameter is set, the value of dfs.namenode.lifeline.handler.count is overwritten. By default, this parameter is not set. If dfs.namenode.lifeline.rpc-address is not configured, this parameter does not take effect.

5. Various Protocols and Services of RPC Server

The \org\apache\hadoop\hdfs\server\namenode\NameNodeRpcServer.java file also describes the protocols and services used by the RPC server.

The protocols and services involved in serviceRpcServer and clientRpcServer are the same. However, serviceRpcServer processes service requests from the server, and clientRpcServer processes service requests from clients.

That's all, thanks!