(Participants need to submit their answers in the required comments of each posts, the replies of the comments will be hidden during the activity.)
1. Definition
SmartDedupe and SmartCompression features are developed by Huawei to provide smart data deduplication and data compression functions.
SmartDedupe is a data downsizing technology that deletes duplicate data blocks in a storage system to save physical storage capacity, meeting growing data storage needs.
SmartCompression reorganizes data to reduce storage space consumption and improve the data transfer, processing, and storage efficiency without any data loss.
The OceanStor V5series storage system supports inline deduplication and compression, that is, only the data written after SmartDedupe and SmartCompression are enabled will be deduplicated and compressed.
When both SmartDedupe and SmartCompression are enabled, deduplication is performed before compression.
2. Basic Concepts
Deduplication data block size: This determines deduplication granularity and is identical to the block size of a thin LUN. To set the block size of a thin LUN, run create lun on the command line interface (CLI) and modify the value of parameter grain_size.
Compression data block size: This determines compression granularity. The storage system compresses data intelligently using the block size of newly-written data in LUNs. Fixed-length deduplication: The storage system deduplicates the data written into a LUN by the specified deduplication data block size. Hash algorithm: This verifies the consistency of data blocks. It computes the fingerprint of a data block, which is a unique binary number with a fixed length. If the fingerprints of two data blocks are the same, the storage system considers them duplicates. Byte-by-byte comparison policy: This is complementary to the hash algorithm. If two data blocks are found to have the same fingerprint, the storage system compares the data blocks byte by byte, ensuring that the deduplication is secure. Deduplication metadata: This stores deduplication information, such as the fingerprints and storage locations of deduplicated data.
3. Working Principle
Deduplication
After SmartDedupe is enabled for a LUN, the OceanStor storage system uses the hash algorithm to calculate the fingerprint of each new data block. It then compares these fingerprints with those of existing data blocks in the LUN. If a new fingerprint is identical to an existing one, it will be deleted and its storage location registered as that of the existing one. If the fingerprint is unique, the new data block is written to disks.
If you have enabled neither SmartDedupe nor SmartCompression when you create a LUN, you cannot enable them any more after the LUN is created.
Figure1 Deduplication process shows the deduplication process.
Figure 1-1 Deduplication process
Figure2 Effects of using SmartDedupe shows the process of writing data to a storage system with SmartDedupe enabled and disabled. In this example, data blocks B and C have identical fingerprints, while data block D has a unique fingerprint.
Figure 1-2 Effects of using SmartDedupe
