Got it

Define respective components of HDFS and YARN and give examples?

Created: Feb 21, 2022 17:08:27Latest reply: Feb 21, 2022 19:12:18 225 3 1 0 0
  HiCoins as reward: 0 (problem unresolved)

Define respective components of HDFS and YARN and give examples?

  • x
  • convention:

Featured Answers
BAZ
MVE Author Created Feb 21, 2022 19:12:18

HDFS COMPONENT 

HDFS component consist of three main components:

1. Name node

2. Data node

3. Secondary Name node

HDFS


Name node: 

It is also known as the master node. It doesn’t stores the actual data or dataset. The basic purpose of Name node is to maintain metadata of all Data node. Metadata basically consist of all the information about Data node where block of data is stored like number of blocks, their location, on which Rack, which Data node the data is stored and other details.


Basic Task of Name node:

· Manage Data node by regular checking its Health.

· Maintaining the metadata of data block stored in Data node

· Regulates client’s access to files.


Data node

It is also known as slave node. It is responsible for storage of Data block. These blocks of data are stored into the Data node as per the availability of storage capacity. Replication of a Data block is stored on different Data node as to avoid total loss of data. Data node sends heartbeat every 3 seconds to Name node as a reminder that the Data node is alive and every 10th heartbeat the signal send to name node by data node is called as Block Report which contain all basic information of that particular data node.


Basic Task of Data node:

· Works as storage for Data blocks.

· It performs various operations like replication and deletion of data as per Name node instructions.


Secondary Name node:

 It is also a master node. The basic function of Secondary Name node is to created the checkpoint of metadata maintained by Name node. It is actually done because in any case if Name node fails than the meta data of whole system can be accessed again which is created as a checkpoint and stored in Secondary Name node.


Basic Function of Secondary Name node:

· Create checkpoint for Name node

· Work as a backup for metadata in case of Name node failure



COMPONENT OF YARN 

Apache Hadoop YARN Architecture consists of the following main components :

  1. Resource Manager: Runs on a master daemon and manages the resource allocation in the cluster.

  2. Node Manager: They run on the slave daemons and are responsible for the execution of a task on every single Data Node.

  3. Application Master: Manages the user job lifecycle and resource needs of individual applications. It works along with the Node Manager and monitors the execution of tasks.

  4. Container: Package of resources including RAM, CPU, Network, HDD etc on a single node.


YARn



View more
  • x
  • convention:

All Answers
ariase88
ariase88 Admin Created Feb 21, 2022 17:09:00

Thanks for contacting the Huawei community!

We are checking your question and will provide an answer to you shortly...
View more
  • x
  • convention:

faysalji
faysalji Moderator Author Created Feb 21, 2022 17:54:05


  1. Hadoop Distributed File System, also known as HDFS, Hadoop architecture, Yet Another Resource Negotiator, also known as YARN, and YARN architecture.




  2. Hadoop Distributed File System (HDFS)

  3. HDFS is a distributed file system that provides access to data across Hadoop clusters. A cluster is a group of computers that work together.


  4. Like other Hadoop-related technologies, HDFS is a key tool that manages and supports analysis of very large volumes petabytes and zetabytes of data.

  5. What is YARN?

  6. YARN is the acronym for Yet Another Resource Negotiator. YARN is a resource manager created by separating the processing engine and the management function of MapReduce.


  7. It monitors and manages workloads, maintains a multi-tenant environment, manages the high availability features of Hadoop, and implements security controls




  8. For more details


  9. https://www.simplilearn.com/hdfs-and-yarn-tutorial



See

https://www.c-sharpcorner.com/interview-question/define-respective-components-of-hdfs-and-yarn

https://www.wikitechy.com/interview-questions/big-data/what-are-the-components-of-hdfs-and-yarn/

https://www.simplilearn.com/tutorials/hadoop-tutorial/what-is-hadoop


View more
  • x
  • convention:

BAZ
BAZ MVE Author Created Feb 21, 2022 19:12:18

HDFS COMPONENT 

HDFS component consist of three main components:

1. Name node

2. Data node

3. Secondary Name node

HDFS


Name node: 

It is also known as the master node. It doesn’t stores the actual data or dataset. The basic purpose of Name node is to maintain metadata of all Data node. Metadata basically consist of all the information about Data node where block of data is stored like number of blocks, their location, on which Rack, which Data node the data is stored and other details.


Basic Task of Name node:

· Manage Data node by regular checking its Health.

· Maintaining the metadata of data block stored in Data node

· Regulates client’s access to files.


Data node

It is also known as slave node. It is responsible for storage of Data block. These blocks of data are stored into the Data node as per the availability of storage capacity. Replication of a Data block is stored on different Data node as to avoid total loss of data. Data node sends heartbeat every 3 seconds to Name node as a reminder that the Data node is alive and every 10th heartbeat the signal send to name node by data node is called as Block Report which contain all basic information of that particular data node.


Basic Task of Data node:

· Works as storage for Data blocks.

· It performs various operations like replication and deletion of data as per Name node instructions.


Secondary Name node:

 It is also a master node. The basic function of Secondary Name node is to created the checkpoint of metadata maintained by Name node. It is actually done because in any case if Name node fails than the meta data of whole system can be accessed again which is created as a checkpoint and stored in Secondary Name node.


Basic Function of Secondary Name node:

· Create checkpoint for Name node

· Work as a backup for metadata in case of Name node failure



COMPONENT OF YARN 

Apache Hadoop YARN Architecture consists of the following main components :

  1. Resource Manager: Runs on a master daemon and manages the resource allocation in the cluster.

  2. Node Manager: They run on the slave daemons and are responsible for the execution of a task on every single Data Node.

  3. Application Master: Manages the user job lifecycle and resource needs of individual applications. It works along with the Node Manager and monitors the execution of tasks.

  4. Container: Package of resources including RAM, CPU, Network, HDD etc on a single node.


YARn



View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.