Got it

Introduction to Kafka

Latest reply: Nov 24, 2021 04:42:32 686 11 12 0 0

Hello everyone, 

I believe you can learn how to deal with Introduction to Kafka from my case.

[Introduction to Kafka]

Kafka is a distributed message publishing and subscription system. It uses a unique design to provide JMS-like features primarily for handling active streaming data. Kafka applies to many scenarios, such as message queue, behavior tracing, O&M data monitoring, log collection, flow processing, event source tracing, and persistent logs.


[Sample Background]

Kafka is a distributed message system. In this system, you can publish and subscribe to messages. Assume that you want to develop a producer to send a message to a topic in the Kafka cluster every second. In addition, you need to implement a consumer to subscribe to the topic. This type of messages are consumed in real time.


[Procedure for Invoking the Windows Environment Sample]


  1. Prepare the environment. For details, see the first lesson.

  2. The difference between the time and the cluster time cannot exceed 5 minutes.

  3. Check whether the C:\Windows\System32\drivers\etc\hosts file contains the domain name and IP address mapping information of all cluster nodes.

  4. Open the kafka-examples directory of the sample code on the IDEA and check the SDK configuration.

  5. By default, the dependency is automatically loaded. If the dependency is not loaded, open the pom.xml file, right-click the file, and choose Add As Maven Project from the shortcut menu. Wait until the project automatically downloads the dependency.

  6. After downloading user authentication credentials from Manager, decompress the user.keytab and krb5.conf files to the src\main\resources directory of the sample code.

  7. Place the consumer.properties,producer.properties,kafkaSecurityMode configuration file in the directory where the client is decompressed, for example, /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_ClientConfig/Kafka/config, to the src\main\resources directory of the sample code.

  8. Modify the USER_KEYTAB_FILE and USER_PRINCIPAL parameters in the src\main\java\com\huawei\bigdata\kafka\example\Producer.java class and src\main\java\com\huawei\bigdata\kafka\example\Consumer.java . Modify LoginUtil.setZookeeperServerPrincipal("zookeeper/hadoop.**") in the securityPrepare method in the src\main\java\com\huawei\bigdata\kafka\example\security\LoginUtil.java class. Set this parameter to the system domain name of the current cluster. You can log in to FusionInsight Manager, choose System > Permissions > Domain and Mutual Trust, and view the value of Local Domain, which is the current system domain name.

  9. Right-click the resources directory and choose Mark Directory as > Resources Root. If yes, ignore it.

  10. Right-click src\main\java\com\huawei\bigdata\kafka\example\Producer.java and run the Run command to start the sample code.

  11. Right-click src\main\java\com\huawei\bigdata\kafka\example\Consumer.java and run the Run command to start the sample code. After the sample is started, perform step 10 synchronously.

[Debugging Procedure in the Linux Environment]

  1. Prerequisites: The cluster client has been installed in the Linux environment.

  2. Completing the Windows Environment Sample Call Procedure

  3. Performing Packaging in a Windows Environment

  4. Run the following maven command to build the sample project dependency JAR package and save it to the lib directory.

  5. mvn dependency:copy-dependencies -DoutputDirectory=lib --settings D:\tools\apache-maven-3.6.3\conf\settings.xml

  6. Check whether the time difference between the Linux environment and the cluster is less than or equal to 5 minutes.

  7. Check whether the JDK version in the Linux environment is 1.8.

  8. Configuring the /etc/hosts File in the Linux Environment

  9. Create a directory in the Linux environment, for example, /opt/kafkatest, and create a subdirectory lib and the same-level directory src/main/resources in the IntelliJ IDEA sample project.

  10. Upload the JAR package in step 3 and the JAR package in the lib directory in step 4 to the lib directory in the Linux environment, for example, /opt/kafkatest/lib.

  11. Copy all files in the src/main/resources directory of the IntelliJ IDEA project to the src/main/resources directory at the same level as the dependent library folder, for example, /opt/kafkatest/src/main/resources.

  12. Check all files in the /opt/kafkatest/src/main/resources directory and the /opt/kafkatest/lib directory and ensure that the current user has the read permission.

  13. Switch to /opt/kafkatest and run the following command to run the sample program:


java -cp .:/opt/kafkatest/lib/*:/opt/kafkatest/src/main/resources com.huawei.bigdata.kafka.example.Producer

I hope it is helpful to you!


  • x
  • convention:

olive.zhao
Admin Created Nov 2, 2021 14:40:27

  • x
  • convention:

SaraZahid
SaraZahid Created Nov 2, 2021 16:18:52 (0) (0)
 
SaraZahid
Created Nov 2, 2021 16:19:25

Introduction to Kafka-4265717-1
View more
  • x
  • convention:

NTan33
Created Nov 3, 2021 02:08:27

A decent introductory walkthrough.
View more
  • x
  • convention:

SamB
Moderator Created Nov 8, 2021 09:57:28

Good introduction...
View more
  • x
  • convention:

faysalji
Moderator Author Created Nov 8, 2021 11:10:04

Good Intro !!!
View more
  • x
  • convention:

Unicef
MVE Created Nov 8, 2021 13:20:18

NICE
View more
  • x
  • convention:

TuanNg
Created Nov 8, 2021 13:57:21

Thanks for sharing
View more
  • x
  • convention:

Majdi.Chebil
Moderator Author Created Nov 8, 2021 18:55:43

Thanks for sharing it!
View more
  • x
  • convention:

Funstuff
Moderator Author Created Nov 9, 2021 11:54:25

Nice introduction
View more
  • x
  • convention:

12
Back to list

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.