Introduction to Kafka

Hello everyone, 

I believe you can learn how to deal with Introduction to Kafka from my case.

[Introduction to Kafka]

Kafka is a distributed message publishing and subscription system. It uses a unique design to provide JMS-like features primarily for handling active streaming data. Kafka applies to many scenarios, such as message queue, behavior tracing, O&M data monitoring, log collection, flow processing, event source tracing, and persistent logs.

[Sample Background]

Kafka is a distributed message system. In this system, you can publish and subscribe to messages. Assume that you want to develop a producer to send a message to a topic in the Kafka cluster every second. In addition, you need to implement a consumer to subscribe to the topic. This type of messages are consumed in real time.

[Procedure for Invoking the Windows Environment Sample]

1. Prepare the environment. For details, see the first lesson.

2. The difference between the time and the cluster time cannot exceed 5 minutes.

3. Check whether the C:\Windows\System32\drivers\etc\hosts file contains the domain name and IP address mapping information of all cluster nodes.

4. Open the kafka-examples directory of the sample code on the IDEA and check the SDK configuration.

5. By default, the dependency is automatically loaded. If the dependency is not loaded, open the pom.xml file, right-click the file, and choose Add As Maven Project from the shortcut menu. Wait until the project automatically downloads the dependency.

6. After downloading user authentication credentials from Manager, decompress the user.keytab and krb5.conf files to the src\main\resources directory of the sample code.

7. Place the consumer.properties,producer.properties,kafkaSecurityMode configuration file in the directory where the client is decompressed, for example, /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_ClientConfig/Kafka/config, to the src\main\resources directory of the sample code.

8. Modify the USER_KEYTAB_FILE and USER_PRINCIPAL parameters in the src\main\java\com\huawei\bigdata\kafka\example\Producer.java class and src\main\java\com\huawei\bigdata\kafka\example\Consumer.java . Modify LoginUtil.setZookeeperServerPrincipal("zookeeper/hadoop.**") in the securityPrepare method in the src\main\java\com\huawei\bigdata\kafka\example\security\LoginUtil.java class. Set this parameter to the system domain name of the current cluster. You can log in to FusionInsight Manager, choose System > Permissions > Domain and Mutual Trust, and view the value of Local Domain, which is the current system domain name.

9. Right-click the resources directory and choose Mark Directory as > Resources Root. If yes, ignore it.

10. Right-click src\main\java\com\huawei\bigdata\kafka\example\Producer.java and run the Run command to start the sample code.

11. Right-click src\main\java\com\huawei\bigdata\kafka\example\Consumer.java and run the Run command to start the sample code. After the sample is started, perform step 10 synchronously.

[Debugging Procedure in the Linux Environment]

1. Prerequisites: The cluster client has been installed in the Linux environment.

2. Completing the Windows Environment Sample Call Procedure

3. Performing Packaging in a Windows Environment

4. Run the following maven command to build the sample project dependency JAR package and save it to the lib directory.

5. mvn dependency:copy-dependencies -DoutputDirectory=lib --settings D:\tools\apache-maven-3.6.3\conf\settings.xml
   
   

6. Check whether the time difference between the Linux environment and the cluster is less than or equal to 5 minutes.

7. Check whether the JDK version in the Linux environment is 1.8.

8. Configuring the /etc/hosts File in the Linux Environment

9. Create a directory in the Linux environment, for example, /opt/kafkatest, and create a subdirectory lib and the same-level directory src/main/resources in the IntelliJ IDEA sample project.

10. Upload the JAR package in step 3 and the JAR package in the lib directory in step 4 to the lib directory in the Linux environment, for example, /opt/kafkatest/lib.

11. Copy all files in the src/main/resources directory of the IntelliJ IDEA project to the src/main/resources directory at the same level as the dependent library folder, for example, /opt/kafkatest/src/main/resources.

12. Check all files in the /opt/kafkatest/src/main/resources directory and the /opt/kafkatest/lib directory and ensure that the current user has the read permission.

13. Switch to /opt/kafkatest and run the following command to run the sample program:

    
    

java -cp .:/opt/kafkatest/lib/*:/opt/kafkatest/src/main/resources com.huawei.bigdata.kafka.example.Producer

I hope it is helpful to you!