Got it

Big data quality

Latest reply: Mar 5, 2022 07:07:39 1301 106 51 0 1

dq



In every data management and analytics project, data quality can be a significant barrier. Typos, various name practices, and data integration challenges can all cause complications. However, data quality is much more important in big data applications since there is a considerably larger amount, diversity, and velocity of data.


Considering big data quality challenges can arise from a variety of applications, data kinds, platforms, and use cases, it is suggested that a 4th V for veracity be included to big data management activities.



Importance


As a result of real-world system results, big data quality flaws can lead to not only erroneous algorithms but also significant accidents and injuries. At the absolute least, corporate users will be less trusting of data sets and the apps developed on top of them. Furthermore, if data accuracy and quality play a role in front-line business decisions, organizations may be exposed to government regulatory inspection.


Only if adequate policies and support structures to regulate and monitor data quality are in place can data become a strategic asset.


Poor-quality data can drive up data management expenses due to the constant cleanup, additional personnel requirements, and compliance difficulties. It can also cause problems with decision-making and planning.



What makes big data different in terms of data quality


For as long as people have been acquiring data, data quality has been a concern. Big data, on the other hand, alters everything.


Consider a team of 100 people that generates and processes a couple gigabytes of client data every day. Managing this volume of data necessitates a new strategy to assuring data quality for big data, which must take into account the following factors:



  • Data forms that are complex and dynamic


    Among event kinds, user groups, program versions, and device types, big data might have various dimensions. Performing checks on specific slices of data, which might easily number in the hundreds or thousands, is required to map out the data quality problem in a meaningful way. When new events and characteristics are added and old ones are deprecated, the structure of data can also change.


  • Issues with scaling


    Import-and-inspect designs that functioned for traditional data files or spreadsheets are no longer practicable. Big data quality practices must be developed for both classic data warehouses and current data lakes, as well as streams of real-time data.


  • Large amount of data


    It's difficult to effectively inspect new data in big data systems. To ensure data quality for big data, quality metrics must be developed that can be tracked automatically against alterations in big data systems and use cases.


  • x
  • convention:

Saqibaz
Created Feb 4, 2022 16:48:04

Thanks for sharing
View more
  • x
  • convention:

smileymind
smileymind Created Feb 9, 2022 04:26:19 (0) (0)
 
user_4358465
user_4358465 Created Feb 28, 2022 04:47:18 (0) (0)
Thank you for taking the time to review this post.  
faysalji
Author Created Feb 8, 2022 05:16:20

Thanks for sharing
View more
  • x
  • convention:

user_4358465
user_4358465 Created Feb 28, 2022 04:47:25 (0) (0)
Thank you for taking the time to review this post.  
zaheernew
MVE Author Created Feb 8, 2022 06:08:01

Useful info
View more
  • x
  • convention:

Sothea_nh
Sothea_nh Created Feb 9, 2022 04:26:52 (0) (0)
 
user_4358465
user_4358465 Created Feb 28, 2022 04:47:32 (0) (0)
Thank you for taking the time to review this post.  
zaheernew
zaheernew Reply user_4358465  Created Feb 28, 2022 04:53:24 (0) (0)
 
Unicef
MVE Created Feb 8, 2022 06:24:17

Thanks for sharing
View more
  • x
  • convention:

Sothea_nh
Sothea_nh Created Feb 9, 2022 04:27:00 (0) (0)
 
user_4358465
user_4358465 Created Feb 28, 2022 04:47:39 (0) (0)
Thank you for taking the time to review this post.  
SamB
Created Feb 8, 2022 09:59:46

thank you for sharing
View more
  • x
  • convention:

user_4358465
user_4358465 Created Feb 28, 2022 04:47:48 (0) (0)
Thank you for taking the time to review this post.  
huyvan
Created Feb 9, 2022 05:05:04

Good share
View more
  • x
  • convention:

user_4358465
user_4358465 Created Feb 28, 2022 04:47:54 (0) (0)
Thank you for taking the time to review this post.  
Faridrami
Created Feb 9, 2022 16:24:32

Thanks for sharing
View more
  • x
  • convention:

shakeela
shakeela Created Feb 9, 2022 16:28:02 (0) (0)
 
user_4358465
user_4358465 Created Feb 28, 2022 04:47:59 (0) (0)
Thank you for taking the time to review this post.  
SaraZahid
Created Feb 9, 2022 16:25:28

Good
View more
  • x
  • convention:

user_4358465
user_4358465 Created Feb 28, 2022 04:48:09 (0) (0)
Thank you for taking the time to review this post.  
SaraZahid
Created Feb 9, 2022 16:25:33

Thanks for sharing
View more
  • x
  • convention:

user_4358465
user_4358465 Created Feb 28, 2022 04:48:15 (0) (0)
Thank you for taking the time to review this post.  

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.