Hello, everyone!
I will share with you how to deal with the problem of full disk when using Kafka. If you can delete the partition directory or segment files in the partition directory, you can perform the following operations.
Solution 1: Delete the entire partition directory.
Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Kafka > Instance to start the stopped Broker instance.
Go to the kafka-logs directory where the disk is full. For example, if the directory is /srv/BigData/kafka/data1, go to the current directory and run the following command to find the top 10 partition directories with the largest capacity in the current disk:
du -s * | sort -nr | head
The output is as follows:
Assume that the partition to be deleted is test_1-5. Perform the following operations:
Delete the entire test_1-5 partition directory.
Modify the replication-offset-checkpoint and recovery-point-offset-checkpoint files in the same directory. The methods for modifying the two files are the same.
After the file is opened, decrease the number in the second line by 1.
Delete the data lines of test_1-5 in the Topic name Distribution ID Offset watermark format, as shown in the following figure.
The data in the figure needs to be deleted. (You are advised not to enter the editing mode. Press d twice to delete the data.)
NOTE:
When modifying the replication-offset-checkpoint and recovery-point-offset-checkpoint files, ensure that no blank line exists between two lines and no space exists at the end of each line.
After modifying replication-offset-checkpoint and recovery-point-offset-checkpoint, save the modification and restart the instance.
Solution 2: Delete some segment files from the partition directory.
Data in Kafka is stored in segments. Deleting an old segment file in a log of Kafka can release some space and retain the latest partition data. Perform the following operations:
Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Kafka > Instance to start the stopped Broker instance.
Go to the kafka-logs directory where the disk is full. For example, if the directory is /srv/BigData/kafka/data1, go to the current directory and run the following command to find the top 10 partition directories with the largest capacity in the current disk:
du -s * | sort -nr | head
The output is as follows:
Assume that the partition to be deleted is test_1-5. Perform the following operations:
a. Go to the test_1-5 directory and run the ll –lrt command to sort the directories by time, as shown in the following figure.
A segment file is a group of file segments whose names end with .timeindex, .log, or .index. Each group of file segments is named in the same format. As shown in the preceding figure, the first file segment is 00000000000392276408, as shown in the following figure:
b. If you want to delete the segment data segments of the 24th and 25th days in a month, run the rm –rf command to delete the file segments of the corresponding time segments. You can also use the Xftp visualization tool to delete the segments, as shown in the following figure.
After logging in to the xftp server, click Modified to sort the time in ascending order. Select the segment of the corresponding time and delete it.
NOTE: Only the earliest data can be deleted.
After the deletion, check whether the disk space is released and whether the data segments of other partitions need to be deleted.
Start the instance.
We warmly welcome you to enjoy our community!
