Got it

[Technical Dry Goods] HBase MOB features

Latest reply: Mar 25, 2022 10:33:39 277 2 3 0 0

Hello, everyone!

Today I'm going to introduce you MRS HBase MOB features.

In practical applications, large and small data, such as image data and documents, need to be stored. Data smaller than 10MB can generally be stored on HBase. For data smaller than 100KB, the read and write performance of HBase is the best. If the data stored in HBase is larger than 100KB or even 10MB in size, insert the same number of data files, but the total amount of data will be large, which will cause frequent compaction and split, occupy a lot of CPU, disk IO frequency is high, and performance Severe decline.

By storing MOB (Medium-sized Objects) data (data from 100KB to 10MB in size) directly in HFile format on a file system (such as HDFS file system), these files are centrally managed through the expiredMobFileCleaner and Sweeter tools, and then these files The address information and size information of is stored as value in the store of ordinary HBase. This can greatly reduce HBase's compression and split frequency and improve performance.

HBase currently enables the MOB function by default. If you need to use the MOB function, users need to specify the use of mob to store data on the specified column family when creating a table or modifying table attributes

In order to enable the HBase MOB function, users need to specify the use of mob to store data on the specified column family when creating a table or modifying table attributes.

Use code to declare the way to use mob storage:


HColumnDescriptor hcd = new HColumnDescriptor("f");
hcd.setMobEnabled(true);


hbase(main):009:0> create 't3',{NAME => 'd', MOB_THRESHOLD => '102400', IS_MOB => 'true'}
 
0 row(s) in 0.3450 seconds
 
=> Hbase::Table - t3
hbase(main):010:0> describe 't3'
Table t3 is ENABLED
 
 
t3
 
 
COLUMN FAMILIES DESCRIPTION
 
 
{NAME => 'd', MOB_THRESHOLD => '102400', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE'
TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW',
IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
 
1 row(s) in 0.0170 seconds
Use the shell to declare the way to use mob, the unit of MOB_THRESHOLD is byte:

Parameter entry

In the MRS Manager system, select "ClusterName of the cluster to be operated > Service> HBase> Configuration", and click "Configure all". Enter the parameter name in the search box.

Table 1 Parameter description

parameter

description

Defaults

hbase.mob.file.cache.size

The number of open file handles in the cache. If the value is set relatively large, the cache can cache more file handles, thereby reducing the frequency of opening and closing files. But if this value is set too much, it will cause too many open file handles. The default value is: "1000" . This parameter is configured on the server ResionServer.

1000

hbase.mob.cache.evict.period

The period for mob cache to reclaim cached mob files is 3600s by default.

3600

hbase.mob.cache.evict.remain.ratio

The proportion of the number of files retained by the mob cache after recycling to the number of cache capacity. When the number of cached files exceeds the value set by hbase.mob.file.cache.size, mob cache recycling will be triggered.

0.5

hbase.master.mob.ttl.cleaner.period

ExpiredMobFileCleanerChore execution cycle, in seconds. The default value is one day (86400 seconds).

Description:

If the time-to-live value expires, that is, the file has been more than 24 hours since it was created, the MOB file will be deleted by the expired mob file cleaning tool.

86400


That's all, thanks!

  • x
  • convention:

olive.zhao
Admin Created Mar 18, 2022 09:38:00

  • x
  • convention:

user_4358465
Created Mar 25, 2022 10:33:39

Nice inroduction and suggestions to MRS HBase MOB features
View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.