MapReduce workload is batch oriented(long streaming reads, large sequential writes), so how can it optimize the network bandwidth usage?

olive.zhao · Created Jul 29, 2022 02:11:05

Hello, friend!

1. The program is the MapReduce service program.

2. After the service program starts to run, it is divided into map task and reduece task map to read the file system data and reduce the output result.

3. Network optimization: MapReduce optimization is complex and needs to be analyzed based on the actual situation. For example, Hadoop decides to split a job into multiple independent map and reduce tasks for execution. It schedules the task and allocates the appropriate resources to it, and decides where to assign a task in the cluster. (If possible, it is usually the location of the data to be processed by the task to minimize network overhead.)

Hope this helps!

ariase88 · Created Jul 28, 2022 16:54:22

Thanks for contacting the Huawei community!

We are checking your question and will provide an answer to you shortly...

olive.zhao · Created Jul 29, 2022 02:11:05

Hello, friend!

1. The program is the MapReduce service program.

2. After the service program starts to run, it is divided into map task and reduece task map to read the file system data and reduce the output result.

3. Network optimization: MapReduce optimization is complex and needs to be analyzed based on the actual situation. For example, Hadoop decides to split a job into multiple independent map and reduce tasks for execution. It schedules the task and allocates the appropriate resources to it, and decides where to assign a task in the cluster. (If possible, it is usually the location of the data to be processed by the task to minimize network overhead.)

Hope this helps!

Saqibaz · Created Jul 29, 2022 06:32:29

Thanks

Jackson.F · Created 7 days ago

Data read/write and data write to disks exert pressure on the network. I think it is not low to increase the bandwidth. I think it is not low to compress data and reduce the number of read and write times on the network. Comprehensive service optimization is required.

MapReduce workload is batch oriented(long streaming reads, large sequential writes), so how can it optimize the network bandwidth usage?

Recommended answer

Third Party’s Trade Secret

`@trans`drinking_poetry`~trans`

My Followers

Enterprise

Consumer

Corporate

Carriers

MapReduce workload is batch oriented(long streaming reads, large sequential writes), so how can it optimize the network bandwidth usage?

Recommended answer

Third Party’s Trade Secret

`@trans`drinking_poetry`~trans`

My Followers