Got it

[FI Components] HA HDFS Architecture

Latest reply: Apr 11, 2022 07:45:33 3962 42 14 0 1

Hello all,

This post mainly talks about the architecture of HDFS. The Hadoop Distributed File System (HDFS) supports high-throughput data access, applies to processing of large data sets, and provides POSIX-like APIs.

The HDFS consists of active and standby NameNodes and multiple DataNodes, as shown in Figure 1.

The HDFS works in master/slave mode. NameNodes run on the master node, and DataNodes run on the slave nodes. ZKFC should run along with the NameNodes.

The NameNodes and DataNodes communicate with each other using Transmission Control Protocol (TCP)/Internet Protocol (IP). The NameNode, DataNode, ZKFC and JournalNode can be deployed on Linux servers.

Figure 1 HA HDFS Architecture

HDFS

Table 1 describes the functions of each module shown in Figure 1.

Table 1 HDFS modules

Module

Function

NameNode

Manages the namespace, directory structure, and metadata of file systems, and provides a backup mechanism.

  • Active NameNode: manages the namespace, directory structure, and metadata of file systems, and records the mapping relationships between data blocks and files to which the data blocks belong.

  • Standby NameNode: synchronizes with active NameNode data, and takes over services from the Active NameNode if the Active NameNode is faulty.

  • Observer NameNode: Its data is synchronized with the data in the Active NameNode, and it processes read requests from the client.

DataNode

Stores data blocks of each file and periodically reports stored data blocks to the NameNode.

JournalNode

Synchronizes metadata between the active and standby NameNodes in the High Availability (HA) cluster.

ZKFC

ZKFC must be deployed for each NameNode. It is responsible for monitoring NameNode status and writing status information to the ZooKeeper. ZKFC also has permission to select the active NameNode.

ZK Cluster

ZooKeeper Cluster is a co-ordination service which helps the ZKFC to perform leader election.

HttpFS gateway

HttpFS is a single stateless gateway process which provides the WebHDFS REST API for external processes and the FileSystem API for the HDFS. HttpFS is used for data transmission between different Hadoop versions. It is also used as a gateway to access the HDFS behind a firewall.


Thanks.

 



The post is synchronized to: FusionInsight Components

  • x
  • convention:

little_fish
Admin Created Mar 5, 2020 06:46:46

Please subscribe to the collection of FI components, https://forum.huawei.com/enterprise/en/forum.php?mod=collection&action=view&ctid=368
View more
  • x
  • convention:

user_3828492
Created Jul 8, 2020 23:20:56

Very useful, thanks for the information
View more
  • x
  • convention:

little_fish
little_fish Created Jul 13, 2020 06:25:28 (0) (0)
 
kasamon
Created Jul 10, 2020 00:41:30

Very useful.
View more
  • x
  • convention:

little_fish
little_fish Created Jul 13, 2020 06:25:42 (0) (0)
thanks  
andrebriceno
Created Jul 11, 2020 22:15:13

Nice job, thank you!
View more
  • x
  • convention:

little_fish
little_fish Created Jul 13, 2020 06:25:54 (0) (0)
Thank you.  
minervaE
Created Jul 13, 2020 23:53:52

Thanks
View more
  • x
  • convention:

user_3828492
Created Aug 10, 2020 20:39:07

Nice job
View more
  • x
  • convention:

user_3828492
Created Aug 11, 2020 18:21:47

Thanks for this presentation
View more
  • x
  • convention:

little_fish
little_fish Created Aug 12, 2020 06:55:13 (0) (0)
 
kasamon
Created Aug 18, 2020 20:14:43

Thanks for sharing
View more
  • x
  • convention:

little_fish
little_fish Created Sep 14, 2020 01:20:42 (0) (0)
cool  
minervaE
Created Sep 10, 2020 21:21:06

Very good
View more
  • x
  • convention:

little_fish
little_fish Created Sep 14, 2020 01:20:52 (0) (0)
 

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.