This article provides an in-depth look into Apache Kafka's fundamental architecture, focusing on its distributed log design, performance optimizations for high throughput and data persistence on HDDs, and its evolution from ZooKeeper to KRaft for distributed consensus. It highlights Kafka's role as a central nervous system for real-time data streaming in large-scale distributed systems.
Read original on High ScalabilityAt the heart of Kafka's design is the immutable, ordered log data structure. This design choice, optimized for linear reads and writes, enables Kafka to achieve high performance even when storing terabytes of data on HDDs. Unlike traditional messaging systems, Kafka persists all data to disk, leveraging the sequential I/O capabilities of HDDs and operating system optimizations like read-ahead and write-behind caching. This allows for O(1) performance for most operations, irrespective of log size, which is critical for systems handling millions of messages per second.
Kafka employs several strategies to achieve its high throughput: grouping messages for network efficiency, batching writes to disk, and utilizing OS pagecache. While zero-copy optimization sounds promising for reducing CPU overhead by directly moving data from pagecache to sockets, its practical impact is often limited in production due to factors like SSL/TLS encryption, which necessitates message modification. The core performance gains come from minimizing random disk I/O and maximizing linear data access.
Kafka operates as a distributed system with nodes called brokers. Topics are divided into partitions, which are replicated across multiple brokers for fault tolerance and availability. Replication is leader-based, meaning a single broker leads a partition, handling all writes and asynchronously replicating to followers. Producers can configure durability via 'acks' settings (0, 1, or 'all'), balancing latency with data safety. Consumers organize into consumer groups, reading from partitions in order and synchronizing their progress via a dedicated '__consumer_offsets' topic and a Group Coordinator.
Key System Design Takeaway
Kafka's decoupling of producers and consumers, enabled by persistent storage and an immutable log, allows for highly scalable and resilient data pipelines. This avoids the coupling issues of traditional message queues where messages are deleted upon consumption, potentially impacting producers during consumer slowdowns.
Historically, Kafka relied on Apache ZooKeeper for distributed consensus, managing metadata like active brokers, topic configurations, and partition assignments. However, Kafka is transitioning to its own Raft-based consensus mechanism called KRaft (Kafka Raft). KRaft extends Kafka's existing replication protocol, treating cluster metadata itself as an immutable log replicated across a quorum of controller brokers. This internalizes consensus, simplifying the architecture and reducing external dependencies.