Got it

Big Data revolution and FusionInsight Highlighted

Latest reply: May 10, 2022 06:15:42 2960 66 21 0 2

Hello, everyone!


We have come across the term 'Big Data' many times, but not many people know what Big Data really is and how it is useful in modern world.


Businesses, government agencies, HCPs (Health Care Providers), as well as financial and academic institutions are all exploiting Big Data's power to boost business prospects and enhance customer experience. 


bd


Every day, the world produces almost 2.5 quintillion bytes of data. In the last two years alone, nearly 90 percent of the global data has been generated.


At this point, we know that Big Data is being utilized by every industry and it is important to know what Big Data really is. Let’s talk about Big Data, its applications and Huawei’s Big Data Solution (FusionInsight).


BIG DATA

The term 'Big Data' refers to information/data that is massive, fast and so challenging that conventional methods make it extremely difficult or impossible to process it.


There are some simple Big Data concepts that will make it much easier to define what Big Data is:


  • it refers to a vast volume of data that in time tends to expand exponentially;


  • it is so extensive that traditional data analysis methods cannot be used to process or evaluate it;


  • data mining, data collection, data processing, data exchange and visualization of data are included in Big Data.


Now that we have a decent idea about Big Data, let’s talk about the types of Big Data.


TYPES OF BIG DATA


There are three types of Big Data:


  • structured;


  • unstructured;


  • semi-structured.


bd


Let me break this down in simple words.


Structured

Structured Data is used in an organized fashion to refer to information that is already stored in databases. We mean that data can be interpreted and saved in a fixed format.


Unstructured

Unstructured data is the opposite of structured data - it doesn’t have a clear format. It makes the collection and analysis of unstructured data very complicated and time-consuming.


Semi-structured

Data that is not structured data in the conventional database format, but includes certain organizational properties that make retrieval simpler, is called semi-structured data.


Assuming you have a good idea about Big Data and it’s types, we’ll now see what Huawei is doing in the market of Big Data and what kind of solution Huawei provides to meet new challenges. Huawei has been in data analytical business for over 12 years and Huawei’s Big Data business has helped hundreds of enterprise customers across all regions. Huawei’s answer to modern world’s big data problems is FusionInsight HD.


FusionInsight HD


Huawei FusionInsight HD is a distributed data processing system that provides massive data analysis and query capabilities. It meets the following requirements of enterprises:


  • swift integration and management of large data sets of various types;


  • advanced analysis of native information;


  • visualization of available data for special analysis;


  • creation of a development environment for new analysis applications;


  • optimization and scheduling of workloads.


FusionInsight is a distributed data-processing system that provides a unified enterprise-level big data storage query and analysis platform by enhancing functions of the open-source Hadoop software.


bd


HADOOP


You must be wondering what is Hadoop, right?


Hadoop is an open source distributed processing framework that manages data processing and storage for big data applications in scalable clusters of computer servers.


Now, Hadoop is open source and its not perfect. It has its own flaws.


So, Huawei adopts essence of the open source Hadoop, eliminates bugs, and improves some functions. Huawei’s FusionInsight is much more stable than open source Hadoop.


It supports the swift integration and management of large dataset of various types, advanced analysis of native information and visualization of available data for special analysis.


With the help of FusionInsight, enterprises can capture new opportunities and discover risks by analyzing and mining various massive data.


FusionInsight ARCHITECTURE


bd


I’ll explain about FusionInsight architecture in details in another article.

 

ADVANTAGES OF FusionInsight


  • Agile;


  • Intelligent;


  •  Convergent.


Agile

  • Provides a range of data processing capabilities, covering converged data warehouse, offline processing, real-time stream computing, real-time retrieval, interactive query, and relationship analysis.


  • Supports unified multi-cluster and multi-tenant management.


  • Supports rolling upgrade with zero downtime.


  • Uses Elk and Spark SQL that are compliant with SQL standards.


Intelligent

  • The graph database responds to correlated data analyses covering tens of billions of records within seconds, promptly returning query results covering hundreds of billions of relationships spanning tens of billions of nodes.


  • RTD enables millisecond-level real-time risk control, making the shift from post-event to real-time risk control.


  • This solution has integrated more than ten algorithms to allow unified algorithm management and improve resource utilization of AI clusters by about 100%.


Convergent

  • Provides DLF for one-stop data integration, development, and management.


  • Converges Hadoop and MPPDB data.


  • Deploys x86 and ARM server hybrid.

 

Below are some technical terminologies which will be useful for engineers.


HDFS

Provides data access with high throughput; can process large-scale data sets.


Yarn

As the resource management system of Hadoop 2.0, Yarn implements resource management and scheduling for applications.


Spark

An in-memory distributed computing framework.


Elk

Provides standard SQL engine and enables conventional applications to be smoothly migrated to the Big Data platform.


MapReduce

A distributed computing engine supporting massive offline batch processing.


Flink

A unified computing framework for batch and stream processing and stream processing. At its core is a stream processing engine that supports data distribution and parallel computing.


Storm

A distributed, reliable, and fault-tolerant real-time stream data processing system. It provides SQL-like query languages (StreamCQL).


Solr

An independent, enterprise-class application search server based on Apache Lucene.


Kafka

A distributed, partitioned message release-subscription system with multiple copies.


Loader

Exchanges data and files between FusionInsight, relational databases, and file systems.


HBase

A column-oriented distributed storage system suitable for mass unstructured or semi-structured data that provides high availability, performance, and scalability. HBase supports real-time data read and write.


Flume

A distributed mass log collection, aggregation, and transmission system that provides high availability and reliability.


GaussDB

Huawei GaussDB integrates AI technology into the database kernel architecture and algorithms, providing users distributed databases with higher performance, higher availability, and more diverse computing power.


SUCCESS CASES OF FusionInsight


  • The Huawei Smart Transportation Solution:


bd


  • Intelligent Marketing with Big Data:


bd


This wraps up my introductory article on Big Data and FusionInsight.


Below you can find all the useful links to learn more about Big Data and FusionInsight.


USEFUL LINKS


1.      https://e.huawei.com/ae/solutions/cloud-computing/big-data;

2.      https://e.huawei.com/en/publications/global/ict_insights/hw_376150/feature story/HW_376292;

3.      https://e.huawei.com/ae/solutions/industries/finance/reshape-data/big-data;

4.      https://www.huaweicloud.com/en-us/solution/government/gbd.html;

5.      https://www.youtube.com/watch?v=v-UzYYjZ4RE.


FusionInsight DOCUMENTATION


https://support.huawei.com/hedex/hdx.do?docid=EDOC1100094385&lang=en&idPath=22658044|7919788|9856606|21110924


The post is synchronized to: FusionInsight Components

  • x
  • convention:

little_fish
Admin Created Nov 17, 2020 00:39:42

Dear,
Here is the link of the FusionInsight topic.
https://forum.huawei.com/enterprise/en/forum.php?mod=collection&action=view&ctid=368
Thank you.
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 04:44:15 (0) (0)
Thank You  
little_fish
little_fish Reply Ihteshamraza  Created Nov 18, 2020 01:00:59 (0) (0)
 
lan2019
lan2019 Created Dec 28, 2020 13:06:53 (0) (0)
Good  
phuta
Created Nov 17, 2020 00:51:35

Thanks for sharing
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 04:44:03 (0) (0)
 
phuta
phuta Reply Ihteshamraza  Created Nov 18, 2020 00:25:25 (0) (0)
 
ethanbrown
ethanbrown Reply phuta  Created Nov 21, 2020 14:27:10 (0) (0)
 
seeratghafoor
Created Nov 17, 2020 04:51:38

great sharing
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 04:52:38 (0) (0)
 
ethanbrown
ethanbrown Reply Ihteshamraza  Created Nov 21, 2020 14:27:19 (0) (0)
 
stephen.xu
Admin Created Nov 17, 2020 04:56:16

Data acquisition and processing are difficult issues.
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 07:12:19 (0) (0)
true, that's why we have FusionInsight and Engineers like you  
wissal
MVE Created Nov 17, 2020 05:05:39

Important knowledge, learned
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 07:12:44 (0) (0)
Thank You friend  
lan2019
lan2019 Created Dec 28, 2020 13:06:26 (0) (0)
Nice  
BAZ
MVE Author Created Nov 17, 2020 06:58:24

Very good information. Thanks for sharing
Can you pls share source of data (graphs) of adoption of big data industry
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 07:13:08 (0) (0)
Sure thing, I'll add it in the post  
Irina
Admin Created Nov 17, 2020 11:25:07

Hmmm
You'd say that the advertising industries would consist of a much higher percentage of Big Data. When in fact, the percentage of telecommunications is much higher Big Data revolution and FusionInsight-3553991-1
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 13:35:07 (0) (0)
I do not disagree with the fact that telecommunication is very important specially for advertising companies, but we should not forget the fact that ways of marketing have totally changed these days. I rarely get calls from companies now. But somehow, whenever I'm planning to buy something, an ad automatically pops up in my social media. This is where big data plays its part.  
Irina
Irina Reply Ihteshamraza  Created Nov 17, 2020 14:12:59 (0) (0)
Good point!  
little_fish
little_fish Reply Ihteshamraza  Created Dec 15, 2020 09:45:13 (0) (0)
I use big data to research Google search.  
DragonVN
Created Nov 17, 2020 13:13:49

Thanks for sharing.
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 17, 2020 13:49:49 (0) (0)
 
anniep
anniep Reply Ihteshamraza  Created Nov 25, 2020 21:57:53 (0) (0)
 
ethanbrown
Created Nov 21, 2020 14:27:24

Thanks for sharing
View more
  • x
  • convention:

Ihteshamraza
Ihteshamraza Created Nov 21, 2020 15:05:18 (0) (0)
thank you  
anniep
anniep Reply Ihteshamraza  Created Nov 25, 2020 21:58:01 (0) (0)
 

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.