Got it

[FI Components] Architecture of Kafka(2)

Latest reply: Nov 24, 2021 04:53:37 1243 14 13 0 1

Hi all,

This post is about the principles in Kafka, and its relationship with other components. 


Principle

  • Message reliability

When a Kafka Broker receives a message, it stores the message on a disk persistently. Each Partition of a Topic has multiple replicas stored on different Broker nodes. If one node is faulty, the replicas on other nodes can be used.

  • High throughput

  1. Kafka provides high throughput in the following ways:

  2. Messages are written into disks instead of being cached in the memory.

  3. The use of Zero-copy eliminates I/O operations.

  4. Data is sent in batches, improving network utilization.

  5. Each Topic is divided into multiple Partitions, which increases concurrent processing. Concurrent read and write operations can be performed between multiple Producers and Consumers. Producers send messages to specified Partitions based on the algorithm used.

  • Message subscribe-notify mechanism

Consumers subscribe to interested Topics and consume data in pull mode. Consumers can choose the consumption mode and control the message pulling speed based on actual situation. Consumers need to maintain the consumption records by themselves.

  • Scalability

When Broker nodes are added to expand the Kafka cluster capacity, the newly added Brokers register with ZooKeeper. After the registration is successful, procedures and Consumers can sense the change in a timely manner and make related adjustment.


Open-source Features

  • Reliability

Message processing methods such as At-Least Once, At-Most Once, and Exactly Once are provided. The message processing status is maintained by Consumers. Kafka needs to work with the application layer to implement the Exactly Once message processing method.

  • High Throughput

High throughput is provided for message publishing and subscription.

  • Persistence

Messages are stored on disks in persistence mode and can be used for batch consumption and real-time application programs. Data persistence and replication prevent data loss.

  • Distribution

A distributed system is easy to be expanded externally. All Producers, Brokers, and Consumers support the deployment of multiple distributed clusters. Systems can be expanded without stopping the running of software or shutting down the machines.


Relationship with Other Components

As a message publishing and subscription system, Kafka provides high-speed data transmission methods for data transmission between different subsystems of the FusionInsight HD platform. Kafka can receive external messages in a real-time manner and provides the messages to the online and offline services for processing. The following figure shows the relationship between Kafka and other components.


Figure 1 Relationship with other components

kafka

Thank you.



The post is synchronized to: FusionInsight Components

  • x
  • convention:

wissal
MVE Created Apr 8, 2020 08:26:25

Well explained, thanks
View more
  • x
  • convention:

little_fish
Admin Created May 19, 2020 06:30:48

Thanks.
View more
  • x
  • convention:

user_4083247
Created Dec 27, 2020 08:08:14

Nice
View more
  • x
  • convention:

little_fish
little_fish Created Apr 8, 2021 02:34:26 (0) (0)
 
Herediano
Created Apr 6, 2021 16:37:00

Thank you
View more
  • x
  • convention:

little_fish
little_fish Created Apr 8, 2021 02:34:19 (0) (0)
 
Zebra
Created Nov 1, 2021 06:09:11

Great share
View more
  • x
  • convention:

little_fish
little_fish Created Nov 1, 2021 09:15:58 (0) (0)
 
kita
Created Nov 2, 2021 03:25:49

Good one
View more
  • x
  • convention:

phuta
Created Nov 4, 2021 08:12:22

Thanks for sharing
View more
  • x
  • convention:

DragonVN
Created Nov 6, 2021 03:30:06

Good share
View more
  • x
  • convention:

bobi
Created Nov 6, 2021 13:24:59

Great share
View more
  • x
  • convention:

12
Back to list

Comment

You need to log in to comment to the post Login | Register
Comment

Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " User Agreement."

My Followers

Login and enjoy all the member benefits

Login

Block
Are you sure to block this user?
Users on your blacklist cannot comment on your post,cannot mention you, cannot send you private messages.
Reminder
Please bind your phone number to obtain invitation bonus.
Information Protection Guide
Thanks for using Huawei Enterprise Support Community! We will help you learn how we collect, use, store and share your personal information and the rights you have in accordance with Privacy Policy and User Agreement.