Got it

Can Data on Kafka Nodes Be Migrated?

Created: May 17, 2022 08:08:06Latest reply: May 17, 2022 08:16:16 153 1 0 0 0
  Rewarded HiCoins: 0 (problem resolved)

Hello everyone!


This post enquires about whether data on Kafka nodes can be migrated. Please check out below.


ISSUE DESCRIPTION


Can data on Kafka nodes be migrated to each other? I would like to migrate partition data between disks on a node without stopping the Kafka service.


Can I do it in Huawei MapReduce?


Thank you very much!

  • x
  • convention:

Featured Answers
olive.zhao
Admin Created May 17, 2022 08:16:16

Yes, Jackson!

You can do it!

I will share with you how to Kafka client commands to migrate partition data between disks on a node without stopping the Kafka service.

1. Log in as a client installation user to the node on which the Kafka client is installed.

2. Run the following command to switch to the Kafka client installation directory, for example,  /opt/kafkaclient:

cd /opt/kafkaclient

3. Run the following command to set environment variables:

source bigdata_env

4. Run the following command to authenticate the user (skip this step in normal mode):

kinit Component service user

5. Run the following command to switch to the Kafka client directory:

cd Kafka/kafka/bin

6. Run the following command to view the topic details of the partition to be migrated:

Security mode:

./kafka-topics.sh --describe --bootstrap-server IP address of the Kafkacluster:21007 --command-config ../config/client.properties --topic topic name

Normal mode:

./kafka-topics.sh --describe --bootstrap-server IP address of the Kafka cluster:21005 --command-config ../config/client.properties --topic Topic name

kafka node

7. Run the following command to query the mapping between Broker_ID and the IP address:

./kafka-broker-info.sh --zookeeper IP address of the ZooKeeper quorumpeer instance:ZooKeeper port number/kafka

8. Obtain the partition distribution and node information from the command output in 6 and 7, and create the JSON file for reallocation in the current directory.

To migrate data in the partition whose Broker_ID is 6 to the /srv/BigData/hadoop/data1/kafka-logs directory, the required JSON configuration file is as follows:

{"partitions":[{"topic": "testws","partition": 2,"replicas": [6,5],"log_dirs": ["/srv/BigData/hadoop/data1/kafka-logs","any"]}],"version":1}

9. Run the following command to perform reallocation:

Security mode:

./kafka-reassign-partitions.sh --bootstrap-server Service IP address of Broker:21007 --command-config ../config/client.properties --zookeeper {zk_host}:{port}/kafka --reassignment-json-file Path of the JSON file compiled in 8 --execute

Normal mode:

./kafka-reassign-partitions.sh --bootstrap-server Service IP address of Broker:21005 --command-config ../config/client.properties --zookeeper {zk_host}:{port}/kafka --reassignment-json-file Path of the JSON file compiled in 8 --execute

If message "Successfully started reassignment of partitions" is displayed, the execution is successful.

Hope this helps!

View more
  • x
  • convention:

All Answers
olive.zhao
olive.zhao Admin Created May 17, 2022 08:16:16

Yes, Jackson!

You can do it!

I will share with you how to Kafka client commands to migrate partition data between disks on a node without stopping the Kafka service.

1. Log in as a client installation user to the node on which the Kafka client is installed.

2. Run the following command to switch to the Kafka client installation directory, for example,  /opt/kafkaclient:

cd /opt/kafkaclient

3. Run the following command to set environment variables:

source bigdata_env

4. Run the following command to authenticate the user (skip this step in normal mode):

kinit Component service user

5. Run the following command to switch to the Kafka client directory:

cd Kafka/kafka/bin

6. Run the following command to view the topic details of the partition to be migrated:

Security mode:

./kafka-topics.sh --describe --bootstrap-server IP address of the Kafkacluster:21007 --command-config ../config/client.properties --topic topic name

Normal mode:

./kafka-topics.sh --describe --bootstrap-server IP address of the Kafka cluster:21005 --command-config ../config/client.properties --topic Topic name

kafka node

7. Run the following command to query the mapping between Broker_ID and the IP address:

./kafka-broker-info.sh --zookeeper IP address of the ZooKeeper quorumpeer instance:ZooKeeper port number/kafka

8. Obtain the partition distribution and node information from the command output in 6 and 7, and create the JSON file for reallocation in the current directory.

To migrate data in the partition whose Broker_ID is 6 to the /srv/BigData/hadoop/data1/kafka-logs directory, the required JSON configuration file is as follows:

{"partitions":[{"topic": "testws","partition": 2,"replicas": [6,5],"log_dirs": ["/srv/BigData/hadoop/data1/kafka-logs","any"]}],"version":1}

9. Run the following command to perform reallocation:

Security mode:

./kafka-reassign-partitions.sh --bootstrap-server Service IP address of Broker:21007 --command-config ../config/client.properties --zookeeper {zk_host}:{port}/kafka --reassignment-json-file Path of the JSON file compiled in 8 --execute

Normal mode:

./kafka-reassign-partitions.sh --bootstrap-server Service IP address of Broker:21005 --command-config ../config/client.properties --zookeeper {zk_host}:{port}/kafka --reassignment-json-file Path of the JSON file compiled in 8 --execute

If message "Successfully started reassignment of partitions" is displayed, the execution is successful.

Hope this helps!

View more
  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.