Flume’s major use case is to gulp down data into Hadoop. Flume is incorporated with Hadoop’s monitoring system, file formats, file system, and utilities such as Morphlines. Along with Flume’s design of sinks, sources, and channels, Flume can help one shift data to other systems lithely. However, the main feature of Hadoop is its Hadoop integration. Flume is the best option to use when we have non-relational data sources or a long file to stream into Hadoop.
On the other hand, Kafka’s major use case is a distributed publish–subscribe messaging system. It is not developed specifically for Hadoop, and using Kafka to read and write data to Hadoop is considerably trickier than it is with Flume. Kafka can be used when we particularly need a highly reliable and scalable enterprise messaging system to connect multiple systems like Hadoop.
