Storage products using erasure coding

2

Erasure coding (EC) is a general algorithm for data distribution and redundancy protection. The application scenarios include storage, arrays, and cloud storage.

Other related questions:
Principles and computing method of Erasure Code for OceanStor 9000
Erasure Code supports two data protection levels: N+M and N+M:B. N+M includes N+1, N+2, N+3, and N+4, and is suitable for scenarios with a large number of nodes. N+M:B includes N+2:1 and N+3:1, and is suitable for scenarios with a small number of nodes. N+M indicates that each data stripe contains N original data strips and M redundant data strips, ensuring high data integrity even when M nodes fail. N+M:B indicates that each data stripe contains N original data strips and M redundant data strips, ensuring high data integrity even when M disks or B nodes fail. Users only need to specify +M or +M:B for directories or files. OceanStor 9000 automatically selects the most appropriate N value based on the number of nodes. N ranges from 2 to 16, and M from 1 to 4. To configure the redundancy ratio: 1. Log in to DeviceManager. 2. Choose Provisioning > Resource Manager. 3. On the page that is displayed, select a directory and click Properties. 4. In the dialog box that is displayed, click the Redundancy Ratio tab and set +RDC/RDN. In N+M scenarios, data slices are stored on N nodes, and redundant data is stored on another M nodes. Among the N+M nodes, up to three are allowed to fail at the same time. However, N is within 2 to 16 and OceanStor 9000 supports a maximum of 288 nodes. Accordingly, some nodes (except the N+M nodes) do not store data slices, and the failure of these nodes, in theory, does not impact on data integrity. In N+M:B scenarios, data slices are stored on some disks of N nodes, and M pieces of redundant data are stored on other disks of the N nodes. The system allows up to M pieces of redundant data to be lost. That is, M disks are allowed to fail at the same time. Similarly, if more than B nodes fail, more than M pieces of redundant data will be lost. Therefore, a maximum of M disks or B nodes are allowed to fail.

Erasure code implementation for different products
Erasure coding (EC) is a general algorithm for data distribution and redundancy protection. The application scenarios include storage, arrays, and cloud storage. Each vendor or product independently implements EC based on common algorithms. What is an erasure code? An erasure code is a general form of the redundant array independent disk (RAID) technology. Any RAID can be converted to a specific erasure code. A conventional RAID supports only a small number of disks and does not meet the requirements of multiple distribution points and nodes. For example, RAID 5 supports a failure of only one disk but RAID 6 supports a failure of only two disks. When multiple disks fail, the erasure coding algorithm provides a solution. Erasure coding (EC) is a technology capable of adding m number of data to n number of raw data and restoring any n number of data in the n+m number of data to raw data. EC includes the encoding and decoding processes. In the encoding process, the n number of raw data increases to n+m number of data which is then stored on different devices. If less than m number of data fails, the data can still be restored using the remaining data.

Code (RS, Raptor, or others) used by Erasure Code of OceanStor 9000
Huawei develops Erasure Code of OceanStor 9000 based on the array code and has the patent and intellectual property of the Erasure Code algorithm.

Storage space usage of OceanStor 9000 when erasure coding technology is used
When the erasure coding (EC) technology is used, OceanStor 9000 calculates the storage space usage based on the following principles: 1. The system security space is 90%. 2. When small files are stored, file metadata takes up about 8% space of OceanStor 9000. When large files are stored, file metadata takes up insignificant space. For example, in 4+2 redundancy mode, the maximum storage space usage is calculated as follows: (1) Small file storage space usage = 90% x (1 �?8%) x 66.7% = 55% (2) Large file storage space usage = 90% x 66.7% = 60%

Computing Erasure Code for OceanStor 9000
Computing Erasure Code for OceanStor 9000: In a normal read/write process, Erasure Code is computed at the CA layer. In a data recovery process, Erasure Code is computed at the OBS layer.

If you have more questions, you can seek help from following ways:
To iKnow To Live Chat
Scroll to top