Hello, everyone!
In this post, I want to share with you the Huawei Big data solution.
Introduction to Huawei Big Data Solution
Kunpeng Big Data Solution
Huawei's secure and controllable Kunpeng Big Data Solution provides one-stop high-performance big data computing and data security capabilities. This solution aims to resolve basic problems such as data security, efficiency, and energy consumption during intelligent big data construction in the public safety industry.
Big Data Pro
This solution adopts the public cloud architecture with storage-compute decoupling, ultimate scalability, and the highest possible efficiency. The highly scalable Kunpeng computing power is used as the computing resource and Object Storage Service (OBS) that supports native multi-protocols is used as the storage pool. The resource utilization of big data clusters can be greatly improved, and the big data cost can be halved.
Drew on HUAWEI CLOUD's extensive experience, Huawei big data solution provides you with high-performance and highly-reliable infrastructure resources and AI training and inference platforms for big data services, witnessing your success in digitization and intelligentization.
Advantages of Huawei Big Data Solution
High security
Controllable servers and big data platforms.
Chip-level data encryption, ensuring data security.
High performance
30% higher performance than common servers of the same level.
Ultimate computing power and high-concurrency application scenario optimization.
Support for big data clusters with 5000+ nodes.
High openness
Compatible with the Arm ecosystem and support for mainstream hardware and software.
OpenLabs for services such as software development, application migration, and compatibility certification.
HUAWEI CLOUD Big Data Services
One-stop service for data development, test, and application.
100% compatibility with open-source ecosystems, 3rd-party components managed as plug-ins, one-stop enterprise platform.
Storage-compute decoupling + Kunpeng optimization for better performance.
HUAWEI CLOUD MRS Overview
MRS is a HUAWEI CLOUD service that is used to deploy and manage the Hadoop system and enables one-click Hadoop cluster deployment.
MRS provides enterprise-level big data clusters on the cloud. Tenants can fully control clusters and easily run big data components such as Hadoop, Spark, HBase, Kafka, and Storm. MRS is fully compatible with open source APIs, and incorporates advantages of HUAWEI CLOUD computing and storage and big data industry experience to provide customers with a full-stack big data platform featuring high performance, low cost, flexibility, and ease-of-use. In addition, the platform can be customized based on service requirements to help enterprises quickly build a massive data processing system and discover new value points and business opportunities by analyzing and mining massive amounts of data in either real time or non-real time.
Advantages of MRS
High performance
Leverages Huawei-developed CarbonData storage technology which allows one data set to apply to multiple scenarios.
Supports such features as multi-level indexing, dictionary encoding, pre-aggregation, dynamic partitioning, and quasi-real-time data query. This improves I/O scanning and computing performance and returns analysis results of tens of billions of data records in seconds.
Supports Huawei-developed enhanced scheduler Superior, which breaks the scale bottleneck of a single cluster and is capable of scheduling over 10,000 nodes in a cluster.
Optimizes software and hardware based on Kunpeng processors to fully release hardware computing power and achieve cost-effectiveness.
Easy O&M
Provides a visualized big data cluster management platform, improving O&M efficiency.
Supports rolling patch upgrade and provides visualized patch release information.
Supports one-click patch installation without manual intervention, ensuring long-term stability of user clusters.
Delivers high availability (HA) and real-time SMS and email notification on all nodes.
High security
With Kerberos authentication, MRS provides role-based access control (RBAC) and sound audit functions.
MRS is a one-stop big data platform that allows different physical isolation modes to be set up for customers in the public resource area and dedicated resource area of HUAWEI CLOUD as well as HCS Online in the customer's equipment room.
A cluster supports multiple logical tenants. Permission isolation enables the computing, storage, and table resources of the cluster to be divided based on tenants.
Cost-effectiveness
Provides various computing and storage choices based on diverse cloud infrastructure.
Separates computing from storage, delivering low-cost massive data storage solutions.
Supports flexible configuration of node and disk specifications.
Supports on-demand scale up or down of cluster.
Supports temporary clusters, which are automatically deleted after job execution.
Supports custom policies and auto scaling of clusters, releasing idle resources on the big data platform for customers.
Application Scenarios of MRS
Offline analysis of massive volumes of data.
Low cost: OBS offers cost-effective storage.
Mass data analysis: Hive analyzes TB/PB-scale data.
Visualized data import and export tool: Loader exports data to Data Warehouse Service (DWS) for business intelligence (BI) analysis.
Large-scale data storage.
Real time: Kafka accesses massive amounts of vehicle messages in real time.
Massive data storage: HBase stores massive volumes of data and supports data queries in milliseconds.
Distributed data query: Spark analyzes and queries massive volumes of data.
Low-latency real-time data analysis.
Real-time data ingestion: Flume implements real-time data ingestion and provides various data collection and storage access methods.
Data source access: Kafka accesses data of tens of thousands of elevators and escalators in real time.
That's all, thanks!
