Introduction to SmartDedupe&SmartCompression

Latest reply: Sep 20, 2019 07:53:07 98 2 3 0

Hello everyone, I'd like to share with you the principle of Huawei's SmartDedupe&SmartCompression feature.


i_f15.gifWhat is SmartDedupe&SmartCompression?

The Data Deduplication feature (SmartDedupe) is a data reduction technology that reduces the amount of physical storage used by data by eliminating redundant blocks in the storage system to meet the growing demand for data storage.

The data compression feature (SmartCompression) refers to the reorganization of data without reducing information, reducing the amount of data to reduce storage space, and improving the transmission, processing and storage efficiency of the storage system.

The purpose and benefits of SmartDedupe & Smart Compression

Purpose and benefit

Detailed description

Reduce the storage space occupied by data.

Using the SmartDedupe&SmartCompression feature, the use of deduplication and data compression technologies alone or in combination can effectively reduce redundant data and reduce the storage space occupied by data.

Reduce the initial procurement cost of the storage system.

Due to the reduced storage space occupied by data, only a small amount of storage devices need to be purchased in the initial stage to meet the data storage needs in the subsequent period of time.

Reduce TCO

Since the number of storage devices purchased is reduced, there is no need to invest a lot of management manpower. At the same time, the operation and maintenance costs of equipment room space, power, cool and maintenance management will also decrease.


Concepts that need to be understood before the SmartDedupe&SmartCompression feature:


Deduplication block size: Used to specify the granularity of the storage system for deduplication. The size of the deduplicated data block is the same as the allocation unit size of the thin LUN. You can run the create lun command in the CLI (Command Line Interface) and set the different allocation unit size for the thin LUN through the grain_size parameter.


Compressed block size: Used to specify the granularity at which the storage system. The storage system performs intelligent merge compression based on the block size of the newly written LUN.


Fixed length deduplication: During the deduplication process, the storage system deduplicates the data written to the LUN according to the same size of the deduplicated data block.


Hash algorithm: A method of verifying the consistency of a block of data. Through this method, the fingerprint information of the data block can be calculated (the fingerprint information is a fixed-length binary value, similar to a human fingerprint, and can be regarded as a unique identifier of each data block). When the fingerprint information of multiple data blocks is consistent, the storage system considers these data blocks to be duplicated.


Byte-by-byte comparison strategy: A supplementary verification method for the hash algorithm when looking for duplicate data blocks. When using a byte-by-byte comparison strategy, the storage system needs to compare the data in the data block byte by byte in addition to the fingerprint.


Deduplication metadata: Saves information about the deduplication process. For example, the fingerprint information of the data block, the storage location of the data after the deduplication, and the like.


The following two features of SmartDedupe&SmartCompression will be introduced regarding data compression features online.


The storage system implements online deduplication and only deduplicates newly written data. After the deduplication function is enabled on the LUN, the storage system compares the newly written data block with the original data block in the LUN. If a duplicate data block is identified, the storage system deletes the newly written data block and the storage location of the deleted data is modified to the location of the original data block in the storage system.

1.  The storage system uses the hash algorithm to calculate the fingerprint information of the newly written data block. The size of the data block is the same as that of the thin LUN. The size of the data block is the same for each deduplication process.


2. Compare the newly written data block with the original data block by fingerprint information.

– Yes => The storage system considers the newly written data block to be duplicated with the original data block, deletes the data block, and points the storage location of the data block to the storage location of the original data block.

– No => The storage system considers the newly written data to be a new data block and writes the new data block to the hard disk. For example, the original data in the LUN is A and B. The application server writes the data block C and the data block D, and the data block C and the data block D are compared with the original data. When different data deduplication strategies are adopted, the data deduplication processing result is shown in the schematic diagram.


a


b


Data Compression


The storage system implements online compression and only compresses newly written data. After the data compression feature is enabled on the LUN, the storage system intelligently combines and compresses each newly written data to improve data compression efficiency. When the storage system performs data compression processing, it will perform different degrees of compression according to the compression policy set by the user. The storage system supports the following two compression strategies.


The Fast: Fast policy is the compression algorithm used by the system by default. The algorithm has a fast compression speed, but the space saving efficiency is lower than that of the Deep strategy.


Deep: The Deep strategy can achieve significant improvements in space savings, but compression and decompression take longer.

c

The SmartDedupe&SmartCompression feature occupies the CPU source for data deduplication and data compression. It is used to calculate fingerprint information, compression, and decompression. Therefore, the more LUNs that are deduplicated and compressed at the same time, the greater the impact on system performance.


SmartDedupe&SmartCompression feature interacts with other features


If the LUN is configured with the SmartDedupe&SmartCompression feature, the LUN does not support the following value-added features: HyperMetro, Snapshot, Clone, LUN Copy, Remote Copy, SmartMigration, Volume Mirror, SmartErase, and SmartTier.


If the LUN is configured with the SmartDedupe&SmartCompression feature, the LUN supports the following value-added features or functions: SmartQoSSmartCacheMapping, and Multi-Tenant.


The eDevLUN hosted from a heterogeneous storage system to the local storage system does not support the SmartDedupe&SmartCompression feature.


After the SmartDedupe&SmartCompression feature is enabled on a thin LUN, the LUN does not support Windows 2012 authentication.


After the SmartDedupe&SmartCompression feature is enabled on a thin LUN, the LUN does not support the VAAI (vStorage APIs for Array Integration) interface.


The SmartDedupe&SmartCompression feature is not supported.


The above content is the Huawei SmartDedupe&SmartCompression features that I want to share with you.

  • x
  • convention:

stephen.xu
Admin Created Sep 20, 2019 07:53:07 Helpful(1) Helpful(1)

Thank you very much
  • x
  • convention:

Hiphop
Hiphop Created Sep 20, 2019 08:12:11
I am very happy that you browse my article  

Reply

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login