Got it

Modifying the Kafka Configuration File

Latest reply: Feb 28, 2022 09:59:22 878 1 2 0 0

You can run the kafka-topics.sh and kafka-configs.sh commands to modify Kafka configurations. The kafka-topics.sh script is used to modify the configuration of a single topic. For details about the configuration that can be modified, see Appendix 1. Kafka-configs.sh allows you to modify topic, broker, user, and client-level configurations. For details about the configurations that can be modified, see Appendix 2.

1. kafka-topics.sh

1. Run the kafka-topics.sh command to add or modify configurations:

kafka-topics.sh --alter --topic <topicName> --zookeeper <ZK service IP:24002/kafka> --config <name=value>

2. The kafka-topics.sh command is as follows:

kafka-topics.sh --describe --topic <topicName> --zookeeper <ZK service IP address: 24002/kafka>

The modified configuration is displayed in Configs.

3. The kafka-topics.sh command is used to delete the configuration.

kafka-topics.sh --alter --topic <topicName> --zookeeper <ZK service IP:24002/kafka> --delete-config <name>

After the configuration is deleted, the modified configuration is also deleted from Configs.

2. kafka-configs.sh

1. Run the kafka-configs.sh command to add or modify topic-level configurations:

kafka-configs.sh --alter --zookeeper <ZK service IP:24002/kafka> --add-config <name=value> --entity-name <topicName > --entity-type topics

You can run the kafka-topics.sh --describe --topic <topicName> --zookeeper <ZK service IP address:24002/kafka> command to check whether the topic-level configuration is modified successfully.

2. Run the kafka-configs.sh command to delete topic-level configurations:

kafka-configs.sh --alter --zookeeper <ZK service IP:24002/kafka> --delete-config <name=value> --entity-name <topicName > --entity-type topics

To check whether the topic-level configuration is deleted successfully, run the kafka-topics.sh --describe --topic <topicName> --zookeeper <ZK service IP address:24002/kafka> command to check whether the deleted configuration exists in Configs.

3. Run the kafka-configs.sh command to modify the broker level configuration.

The command for modifying the configuration of a single broker is as follows:

kafka-configs.sh --alter --bootstrap-server <kafka service IP:port> --add-config <name=value> --entity-name <brokerID> --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

Run the following command to modify the configuration of all brokers:

kafka-configs.sh --alter --bootstrap-server <kafka service IP:port> --add-config <name=value> --entity-default --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

Note: If the name in the config file is a configuration that does not exist on the broker, the command can be executed successfully. However, the corresponding configuration is null.

4. Run the kafka-configs.sh command to view the broker-level configuration.

Run the following command to view the configuration modification of a single broker:

kafka-configs.sh -- describe --bootstrap-server <kafka service IP:port> --entity-name <brokerID> --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

Run the following command to view the modified configuration of all brokers:

kafka-configs.sh -- describe --bootstrap-server <kafka service IP:port> --entity-default --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

5. Run the kafka-configs.sh command to delete the broker-level configuration.

To delete a broker, run the following command:

kafka-configs.sh --alter --bootstrap-server <kafka service IP:port> --delete-config <name> --entity-name <brokerID> --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

Run the following command to delete all brokers:

kafka-configs.sh --alter --bootstrap-server <kafka service IP:port> --delete-config <name> --entity-default --entity-type brokers --command-config <property file>. The protocol corresponding to security.protocol in the property file must be the same as the used port. (The command-config parameter is not required when port 21005 is used.)

Appendix I:

The topic-level configurations supported by kafka-topics.sh are modified as follows:

cleanup.policy

compression.type

delete.retention.ms

file.delete.delay.ms

flush.messages

flush.ms

follower.replication.throttled.replicas

index.interval.bytes

leader.replication.throttled.replicas

log.partition.strategy

max.message.bytes

message.format.version

message.timestamp.difference.max.ms

message.timestamp.type

min.cleanable.dirty.ratio

min.compaction.lag.ms

min.insync.replicas

preallocate

retention.bytes

retention.ms

segment.bytes

segment.index.bytes

segment.jitter.ms

segment.ms

unclean.leader.election.enable

Appendix II:

The topic-level configurations supported by kafka-configs.sh are modified as follows:

cleanup.policy

compression.type

delete.retention.ms

file.delete.delay.ms

flush.messages

flush.ms

follower.replication.throttled.replicas

index.interval.bytes

leader.replication.throttled.replicas

log.partition.strategy

max.message.bytes

message.format.version

message.timestamp.difference.max.ms

message.timestamp.type

min.cleanable.dirty.ratio

min.compaction.lag.ms

min.insync.replicas

preallocate

retention.bytes

retention.ms

segment.bytes

segment.index.bytes

segment.jitter.ms

segment.ms

unclean.leader.election.enable

The broker-level configuration supported by kafka-configs.sh is modified as follows:

log.message.timestamp.type

ssl.client.auth

log.retention.ms

sasl.kerberos.ticket.renew.window.factor

log.preallocate

log.index.size.max.bytes

ssl.truststore.type

ssl.keymanager.algorithm

log.cleaner.io.buffer.load.factor

ssl.key.password

background.threads

log.retention.bytes

ssl.trustmanager.algorithm

log.segment.bytes

log.cleaner.delete.retention.ms

log.segment.delete.delay.ms

min.insync.replicas

ssl.keystore.location

ssl.cipher.suites

log.roll.jitter.ms

log.cleaner.backoff.ms

sasl.jaas.config

principal.builder.class

log.flush.interval.ms

log.cleaner.dedupe.buffer.size

log.flush.interval.messages

advertised.listeners

num.io.threads

listener.security.protocol.map

sasl.enabled.mechanisms

ssl.truststore.password

listeners

metric.reporters

ssl.protocol

sasl.kerberos.ticket.renew.jitter

ssl.keystore.password

sasl.mechanism.inter.broker.protocol

log.cleanup.policy

sasl.kerberos.principal.to.local.rules

sasl.kerberos.min.time.before.relogin

num.recovery.threads.per.data.dir

log.cleaner.io.max.bytes.per.second

log.roll.ms

ssl.endpoint.identification.algorithm

unclean.leader.election.enable

message.max.bytes

log.cleaner.threads

log.cleaner.io.buffer.size

sasl.kerberos.service.name

ssl.provider

follower.replication.throttled.rate

log.index.interval.bytes

log.cleaner.min.compaction.lag.ms

log.message.timestamp.difference.max.ms

ssl.enabled.protocols

log.cleaner.min.cleanable.ratio

replica.alter.log.dirs.io.max.bytes.

per.second

ssl.keystore.type

ssl.secure.random.implementation

ssl.truststore.location

sasl.kerberos.kinit.cmd

leader.replication.throttled.rate

num.network.threads

compression.type

num.replica.fetchers

The user-level configurations supported by kafka-configs.sh are modified as follows:

request_percentage

producer_byte_rate

SCRAM-SHA-256

SCRAM-SHA-512

consumer_byte_rate

The client-level configurations supported by kafka-configs.sh are modified as follows:

request_percentage

producer_byte_rate

consumer_byte_rate

The topics_limit configuration supported by kafka-configs.sh is modified as follows:

producer_byte_rate

consumer_byte_rate

Note 1: 

The parameters in Appendix 1 and Appendix 2 are supported by the 651X. For versions earlier than 651X, you can run the kafka-topics.sh and kafka-config.sh commands to view the parameters in --config and --add-config.

Note 2:

 When you use kafka-config.sh to modify num.io.threads and num.network.threads, you can only increase the value within 2 times of the original value or decrease the value within 1/2 times of the original value. Otherwise, an error is reported. For example, the default value of num.io.threads in the FusionInsight is 8. If the default value is changed to 24, the following error message is displayed:

To change the value of num.io.threads to 24, run the following command twice to change the value to 16 and then to 24.


  • x
  • convention:

olive.zhao
Admin Created Feb 28, 2022 09:59:22

  • x
  • convention:

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.