Got it

HDFS RPC Server Summary

Latest reply: Feb 19, 2022 05:21:45 4117 8 8 0 0

Hello, everyone!

The post will share with you the HDFS RPC server.

1. RPC Server Type

The following table lists the RPC server types and related configuration parameters in HDFS. The \org\apache\hadoop\hdfs\server\namenode\NameNodeRpcServer.java file describes declaration and initial configuration of the RPC server.

RPC Type

Parameter

Default Value

lifelineRpcServer

dfs.namenode.lifeline.rpc-bind-host

 

dfs.namenode.lifeline.rpc-address

 

dfs.namenode.lifeline.handler.ratio

0.1f

dfs.namenode.lifeline.handler.count

Math.max((int)(handlerCount * lifelineHandlerRatio), 1);

clientRpcServer

dfs.namenode.rpc-bind-host

 

dfs.namenode.rpc-address

 

dfs.namenode.handler.count

10

serviceRpcServer

dfs.namenode.servicerpc-bind-host

 

dfs.namenode.servicerpc-address

 

dfs.namenode.service.handler.count

10

 

2. clientRpcServer Parameter Description

2.1. dfs.namenode.rpc-bind-host

Specify the actual address bound to the RPC server. If this address is set, the host name part of dfs.namenode.rpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

2.2. dfs.namenode.rpc-address

Specify the RPC address for processing all client requests. If there are multiple NameNode processes for HA/Federation, you need to add the NameService ID to the parameter name, for example, dfs.namenode.rpc-address.ns1. The value is in the format of nn-host1:rpc-port.

2.3. dfs.namenode.handler.count

Listen to the number of NameNode RPC server threads of requests from clients. If dfs.namenode.servicerpc-address is not configured, the NameNode RPC server threads need to listen to requests from all nodes.

3. serviceRpcServer Parameter Description

3.1. dfs.namenode.servicerpc-bind-host

Specify the actual address bound to the service RPC server. If this parameter is set, the host name part of dfs.namenode.servicerpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

3.2. dfs.namenode.servicerpc-address

Specify the RPC address used for communication with the HDFS service. If the parameter is set, BackupNode, Datanodes, and all other services are connected to the address. If there are multiple NameService processes for HA/Federation, you need to add the NameService ID to the parameter name, for example, dfs.namenode.servicerpc-address.ns1. The value is in the format of nn-host1:rpc-port. If the parameter is not set, the value of dfs.namenode.rpc-address is used as the default value.

3.3. dfs.namenode.service.handler.count

Listen to the number of NameNode RPC server threads of requests from DataNode and other non-client nodes. dfs.namenode.service.handler.count is valid only when dfs.namenode.servicerpc-address is configured.

4. lifelineRpcServer Parameter Description

4.1. dfs.namenode.lifeline.rpc-bind-host

Specify the actual address bound to the lifeline RPC server. If this address is set, the host name part of dfs.namenode.lifeline.rpc-address will be overwritten. It can also specify NameNode or NameService for HA/Federation. If this parameter is set to 0.0.0.0, NameNode listens to all interfaces.

4.2. dfs.namenode.lifeline.rpc-address

Specify the NameNode lifeline RPC address. This is an optional RPC address, which can be used to perform health check and node status reporting, thus preventing resource exhaustion in the main RPC processing program pool. If there are multiple NameNode processes for HA/Federation, you need to add the NameSer ID to the parameter name, for example, dfs.namenode.lifeline.rpc-address.ns1. The value is in the format of nn-host1:rpc-port. If this parameter is not set, NameNode does not start the lifeline RPC server. By default, this parameter is not set.

4.3. dfs.namenode.lifeline.handler.ratio

Specify the ratio of dfs.namenode.lifeline.handler.count to dfs.namenode.handler.count. The value of dfs.namenode.handler.count is the function of the number of DataNode nodes in the cluster. This parameter is used to automatically adjust the number of threads processed by the lifeline RPC server. As the message processing is lightweight, the number of threads processed by the lifeline RPC server is less than that required by the active NameNode RPC server. If dfs.namenode.lifeline.handler.count is configured, this parameter is not used. dfs.namenode.lifeline.handler.count specifies the absolute number of threads. If dfs.namenode.lifeline.rpc-address is not configured, this parameter does not take effect.

4.4. dfs.namenode.lifeline.handler.count

Specify the absolute number of threads running on the NameNode lifeline RPC server. These threads are used to process the lifeline protocol of DataNode and HA health check requests from ZKFC. If this parameter is set, the value of dfs.namenode.lifeline.handler.count is overwritten. By default, this parameter is not set. If dfs.namenode.lifeline.rpc-address is not configured, this parameter does not take effect.

5. Various Protocols and Services of RPC Server

The \org\apache\hadoop\hdfs\server\namenode\NameNodeRpcServer.java file also describes the protocols and services used by the RPC server.

The protocols and services involved in serviceRpcServer and clientRpcServer are the same. However, serviceRpcServer processes service requests from the server, and clientRpcServer processes service requests from clients.

RPC Type

Applicable Protocols and Services

serviceRpcServer

ClientNamenodeProtocolPB

HAServiceProtocolPB: haPbService

NamenodeProtocolPB: NNPbService

DatanodeProtocolPB: dnProtoPbService

RefreshAuthorizationPolicyProtocolPB: refreshAuthService

RefreshUserMappingsProtocolPB: refreshUserMappingService

RefreshCallQueueProtocolPB: refreshCallQueueService

GenericRefreshProtocolPB: genericRefreshService

GetUserMappingsProtocolPB: getUserMappingService

TraceAdminProtocolPB: traceAdminService

clientRpcServer

ClientNamenodeProtocolPB

HAServiceProtocolPB: haPbService

NamenodeProtocolPB: NNPbService

DatanodeProtocolPB: dnProtoPbService

RefreshAuthorizationPolicyProtocolPB: refreshAuthService

RefreshUserMappingsProtocolPB: refreshUserMappingService

RefreshCallQueueProtocolPB: refreshCallQueueService

GenericRefreshProtocolPB: genericRefreshService

GetUserMappingsProtocolPB: getUserMappingService

TraceAdminProtocolPB: traceAdminService

lifelineRpcServer

HAServiceProtocolPB

DatanodeLifelineProtocolPB: lifelineProtoPbService

That's all, thanks!

  • x
  • convention:

little_fish
Admin Created Sep 29, 2019 07:50:15

Remote procedural call (RPC) server
View more
  • x
  • convention:

simchamnan
simchamnan Created Nov 21, 2021 22:14:12 (0) (0)
 
stephen.xu
Admin Created Dec 18, 2019 07:38:34

HDFS RPC Server

That is very smart!
View more
  • x
  • convention:

futurework
Created Oct 15, 2021 16:02:49

GOOD
View more
  • x
  • convention:

Unicef
MVE Created Nov 22, 2021 12:07:21

Good share thanks
View more
  • x
  • convention:

csk99
Created Feb 1, 2022 07:58:24

Very Good Post
View more
  • x
  • convention:

olive.zhao
olive.zhao Created Feb 16, 2022 00:43:51 (0) (0)
Yes, I think so, too.  
user_4358465
Created Feb 19, 2022 05:21:45

Informative post, thank you!
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.