Definition of Flume

5

Flume is a distributed, highly reliable, and HA massive log aggregation system. It supports custom data transmitters for data collection. Flume also processes data roughly and writes data to customizable receivers.

Other related questions:
What is the concept of Flume?
Flume is a distributed, highly reliable, and HA massive log aggregation system. It supports custom data transmitters for data collection. Flume also processes data roughly and writes data to customizable receivers.

Flume structure
Flume Agent consists of Source, Channel, and Sink. A Source receives data or generates data through a special mechanism, and sends the data to one or more Channels in batches. There are two types of Sources, namely, data-driven and polling. A Channel is located between a Source and a Sink and used to cache the received data. After the Sink successfully sends the data to the next-hop Channel or the final destination, the data is removed from this Channel. A Sink is responsible for sending data to the next hop or final destination and removing the data from the Channel after sending the data.

Flume Source types
A Source must be associated with at least one Channel. Typical Source types are as follows: 1. Sources that are integrated with the system, such as Syslog and Netcat 2. Sources that automatically generate events, such as Exec and SEQ 3. IPC Sources used for communication between Agents, such as Avro

What is Flume used for?
Flume is a distributed, highly reliable, and HA massive log aggregation system. It supports custom data transmitters for data collection. Flume also processes data roughly and writes data to customizable receivers.

Flume Sink types
A Sink must function with a specific Channel. Typical Sink types are as follows: 1. Sinks that send stored data to the final destination, such as HDFS and HBase 2. Sinks that are consumed automatically, for example, Null Sink 3. IPC Sinks used for communication between Agents, such as Avro

If you have more questions, you can seek help from following ways:
To iKnow To Live Chat
Scroll to top