⚡ Apache Kafka & Real-Time Event Streaming
1️⃣ Traditional Queues vs Distributed Event Streams
Definition: Apache Kafka ek horizontally-scalable, fault-tolerant distributed event store aur streaming platform hai. Yeh messages ko volatile buffer me rakhkar delete karne ke bajay disk-backed immutable append-only commit log ke roop me persist karta hai.
| Criteria | Traditional Message Queue (RabbitMQ / SQS) | Apache Kafka (Event Streaming) |
|---|---|---|
| Message Consumption | Message consume hote hi queue se delete (ACK) ho jata hai. | Immutable Log: Read hone ke baad delete nahi hota; multiple independent consumers purane events dubara replay kar sakte hain. |
| Throughput | Thousands of msgs/sec (Routing logic and consumer ACKs heavy hote hain). | Millions of msgs/sec: Zero-copy OS calls (sendfile) aur disk sequential I/O. |
| Ordering Guarantee | Queue level par global FIFO order maintain karna difficult hota hai scale par. | Partition Level Strict Ordering: Ek partition ke andar events strictly sequential hote hain. |
2️⃣ Core Architecture: Brokers, Topics & Partitions
Kafka data ko logical Topics me organize karta hai, jo horizontally scalable Partitions me split hote hain:
Fundamental Hierarchy:
• Broker: Single Kafka server node jo partitions store karta hai aur client requests serve karta hai.
• Topic: Category name jisme producers records publish karte hain (e.g.
• Partition: Topic ka physical slice jo disk par ek append-only log file hota hai. Har record ko ek strictly incremental number milta hai jise Offset kehte hain.
• Replication Factor (RF): Har partition ka ek Leader hota hai aur multiple Followers (In-Sync Replicas - ISR) hote hain high availability ke liye.
• Broker: Single Kafka server node jo partitions store karta hai aur client requests serve karta hai.
• Topic: Category name jisme producers records publish karte hain (e.g.
user-signups).• Partition: Topic ka physical slice jo disk par ek append-only log file hota hai. Har record ko ek strictly incremental number milta hai jise Offset kehte hain.
• Replication Factor (RF): Har partition ka ek Leader hota hai aur multiple Followers (In-Sync Replicas - ISR) hote hain high availability ke liye.
3️⃣ Producer Internals: Batching & ACKs
Producers messages ko network par single-single bhejne ke bajay micro-batches me bundle karte hain:
// Essential Kafka Producer Configurations (Java / Go / Python)
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker-1:9092,kafka-broker-2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// 1. Durability Acknowledgment Settings
props.put(ProducerConfig.ACKS_CONFIG, "all"); // Leader + all ISR replicas must confirm write
// 2. Throughput Optimization (Batching)
props.put(ProducerConfig.LINGER_MS_CONFIG, 20); // Wait up to 20ms to fill batch
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024); // 64 KB buffer batch size
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy"); // Fast compression algorithm4️⃣ Consumer Groups & Partition Rebalancing
Kafka high-scale horizontal consumption Consumer Groups ke zariye achieve karta hai:
| Scenario (Topic with 4 Partitions) | Consumer Allocation | Operational Behavior |
|---|---|---|
| Group with 2 Consumers | C1 → [P0, P1], C2 → [P2, P3] | Workload evenly distributed 50-50%. |
| Group with 4 Consumers | C1 → P0, C2 → P1, C3 → P2, C4 → P3 | Ideal 1:1 max concurrency alignment. |
| Group with 6 Consumers | 4 Active, 2 Idle | 2 consumers completely idle baithenge (1 partition can only have 1 consumer per group). |
Live Kafka Partition Append & Consumer Group Simulator
[Kafka Cluster]: Topic: 'job-applications' (3 Partitions: P0, P1, P2) | Consumer Group: 'notification-service' (2 Consumers). Ready...
5️⃣ Exactly-Once Semantics (EOS) & Transactions
Distributed streaming me standard delivery guarantees:
- At-Most-Once: Offsets message process hone se pehle commit ho jaate hain. System crash hone par data loss ka risk hota hai (Zero duplicates).
- At-Least-Once: Offsets processing complete hone ke baad commit hote hain. Crash par message dubara consume hoga (Possible duplicates).
- Exactly-Once Processing (EOS):
enable.idempotence=true+ Kafka Transactions. Network retries par broker duplicate PID (Producer ID) aur Sequence Numbers discard kar deta hai.
// Transactional Producer Workflow (Atomic Multi-Partition Writes)
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("order-events", orderId, orderPayload));
producer.send(new ProducerRecord<>("audit-logs", orderId, "Order placed successfully"));
producer.commitTransaction(); // Both published atomically!
} catch (Exception e) {
producer.abortTransaction(); // Rollback stream uncommitted buffers
}6️⃣ Retention Policies & Compacted Topics
| Cleanup Policy | Config Property | Behavior / Use Case |
|---|---|---|
| Delete (Time/Size based) | log.cleanup.policy=delete |
Events 7 din (log.retention.hours=168) ya max size cross karne par hard delete ho jaate hain. Web analytics logs ke liye best. |
| Compact (Key-based) | log.cleanup.policy=compact |
Kafka purane duplicate keys ko clean karta hai aur har unique key ka latest known value state preserve rakhta hai (Database CDC streams). |
7️⃣ KRaft Consensus & Production Architecture
Modern Kafka me Apache ZooKeeper ko completely replace kar diya gaya hai KRaft (Kafka Raft Metadata Mode) ke zariye:
# KRaft Architecture Benefits:
1. Millions of Partitions Scalability: ZooKeeper bottleneck eliminated.
2. Sub-second Controller Failover: Metadata directly broker internal log me rehta hai.
3. Simplified Ops: Sirf ek single binary process manage karni hoti hai.
Production Checklist:
• Partition count calculate karein: $\text{Partitions} = \max(\frac{\text{Target Throughput}}{\text{Producer Throughput}}, \frac{\text{Target Throughput}}{\text{Consumer Throughput}})$.
• Minimum In-Sync Replicas:
• Disk Setup: Fast NVMe SSDs configure karein sequential write performance maintain rakhne ke liye.
• Partition count calculate karein: $\text{Partitions} = \max(\frac{\text{Target Throughput}}{\text{Producer Throughput}}, \frac{\text{Target Throughput}}{\text{Consumer Throughput}})$.
• Minimum In-Sync Replicas:
min.insync.replicas=2 (with RF=3) ensure karein data loss prevent karne ke liye.• Disk Setup: Fast NVMe SSDs configure karein sequential write performance maintain rakhne ke liye.
