Do deduplication and compression delete valid data?

Dear Lucas,

We warmly remind you that the deduplication technology divides data into blocks of a fixed size (for example, 4 KB/8 KB) and calculates fingerprints using the Hash algorithm. If the fingerprint and data are the same as existing ones, the system increases the fingerprint reference count and does not flush duplicate data to disks. The compression technology searches for the longest identical fields in a certain range in the unit of byte. The system re-organizes data using the specified coding formats to reduce physical space occupied by the data. Therefore, deduplication and compression delete redundant data and retain valid data. It’s my pleasure to help you.

Hi there!

What little_fish said.

Specifically: When the system finds duplicates in the fingerprint database, the blocks that it suspects to be duplicate (SHA1 algorythm can actually generate collisions against a single occurrence, collisions are highly unlikely, but, for instance, when providing for a hot database, potentially bilions of blocks could change simultaneously, so the chance that the SHA1 algorythm will make a mistake (collision) is low, but still real. So, to make absolutely completely sure, the system will compare the 8 KB blocks to eachother byte against byte. (1000 comparisons of 8 bits each) The chance of making a mistake during this comparison is so close to zero, that we can safely use this method to deduplicate data. This is to make absolutely sure that the system won't delete or flush data that is unique. So... no. Deduplication and compression do not delete valid data in AFA's. Not even in hybrid systems, like OceanStor V3, or V5 (dedupe works the same there)

@little_fish: I see a lot of your posts on the site. I like them a lot :)

Thanks. Let’s build a better community together!

Hi

Look at this document from Huawei, It has detailed explanation on how deduplication and compression works with flowcharts and diagrams. Hope this helps!!

Refer this Document OceanStor Dorado V6 6.0.0 SmartDedupe and SmartCompression from https://support.huawei.com/enterprise

Link:

https://support.huawei.com/enterprise/en/doc/EDOC1100122862?idPath=7919749|7941815|21430818|21462743|24030083

@Lucas_Zhao