Safety Instruction on Using Kafka

Hello, everyone! 

I will share with you how to deal with the problem of safety instruction on using kafka. 

Scenario

Kafka provides secure access interfaces since C60SPC200. However, to ensure compatibility, Kafka still provides non-secure access interfaces.

• Brief Introduction to Kafka APIs

• Protocol Description for Accessing Kafka

• ACL Settings for Topics

• Use of New and Old Kafka APIs in Different Scenarios

Brief Introduction to Kafka APIs

• New producer API

  Indicates the API defined in org.apache.kafka.clients.producer.KafkaProducer. When kafka-console-producer.sh is used, the API is used by default.

• Old producer API

  Indicates the API defined in kafka.producer.Producer. When kafka-console-producer.sh is used, the API is invoked to add --old-producer.

• New consumer API

  Indicates the API defined in org.apache.kafka.clients.consumer.KafkaConsumer. When kafka-console-consumer.sh is used, the API is invoked to add --new-consumer.

• Old consumer API

  Indicates the API defined in kafka.consumer.ConsumerConnector. When kafka-console-consumer.sh is used, the API is used by default.

NOTE: New producer API and new consumer API are called new API in general in the document.

Protocol Description for Accessing Kafka

The protocols used to access Kafka are as follows: PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL.

When the Kafka service is started, the listeners using the PLAINTEXT and SASL_PLAINTEXT protocols are started by default. You can set ssl.mode.enable to true in Kafka service configuration to start listeners using SSL and SASL_SSL protocols.

The following table describes the four protocols:

ACL Settings for Topics

Kafka supports secure access. Therefore, users can set the ACL for topics to ensure that different users access different topics. To view and set the permission information about a topic, run the kafka-acls.sh script on the Linux client.

• Scenario

  Assign Kafka users with specific permissions for related topics based on service requirements.

  The following table describes default Kafka user groups.

  User Group	Description
  kafkaadmin	Indicates the Kafka administrator user group. Users in this group are eligible to create, delete, authorize, as well as read data from and write data to a topic.
  kafkasuperuser	Indicates a super Kafka user group. Users in this group are eligible to read data from and write data to all topics.
  kafka	Indicates a common Kafka user group. If users in this group want to read data from and write data to a specific topic, the users in the kafkaadmin group must grant permissions to users in this group.

• Prerequisites

1. The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the kafkaadmin group).

2. The Kafka client has been installed.

• Procedur

1. Use PuTTY to log in to the node where the client is installed as the Kafka client installation user.

2. Run the following command to switch to the Kafka client installation directory, for example, /opt/kafkaclient:

   cd /opt/kafkaclient

3.  Run the following command to configure environment variables:

   source bigdata_env

4. Run the following command to perform user authentication (skip this step in normal mode):

   kinit component service user

5. Run the following command to go to the Kafka client installation directory:

   cd Kafka/kafka/bin

6. The following describes the commands commonly used for user authorization when kafka-acl.sh is used:

• View the permission control list of a topic:

  ./kafka-acls.sh --authorizer-properties zookeeper.connect=<Service IP address of ZooKeeper clusters:24002/kafka > --list --topic <Topic name>

• Add the producer permission for a user:

  ./kafka-acls.sh --authorizer-properties zookeeper.connect=<Service IP address of ZooKeeper clusters:24002/kafka > --add --allow-principal User:<username> --producer --topic <Topic name>

• Remove the producer permission from a user:

  ./kafka-acls.sh --authorizer-properties zookeeper.connect=<Service IP address of ZooKeeper clusters:24002/kafka > --remove --allow-principal User:<username> --producer --topic <Topic name>

• Add the consumer permission for a user:

  ./kafka-acls.sh --authorizer-properties zookeeper.connect=<Service IP address of ZooKeeper clusters:24002/kafka > --add --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>

• Remove the consumer permission from a user:

  ./kafka-acls.sh --authorizer-properties zookeeper.connect=<Service IP address of ZooKeeper clusters:24002/kafka > --remove --allow-principal User:<username> --consumer --topic <Topic name> --group <consumer group name>

  
  

We warmly welcome you to enjoy our community!