Messaging, Events and Email Systems · Event-Driven Architecture
Kafka in Practice
Topics, partitions, consumer groups, keys and retention.
Kafka topics are split into partitions; messages with the same key go to the same partition, preserving their order. Consumer groups share partitions — parallelism is capped by the partition count. Consumers commit offsets; rebalances can cause redelivery. Configure replication (RF=3, min.insync.replicas=2, acks=all), choose retention (time/size) or compaction (latest value per key), and size partitions for future throughput.
# Producer (reliable)
acks=all
enable.idempotence=true
compression.type=zstd
# Topic
kafka-topics --create --topic order-events --partitions 12 --replication-factor 3 \
--config min.insync.replicas=2 --config retention.ms=604800000
# Consumer
group.id=fulfillment
enable.auto.commit=false # commit after processing
max.poll.interval.ms=300000
Common pitfalls
- Too few partitions to scale consumers later (repartitioning reorders keys).
- Hot keys overloading one partition.
- Slow processing exceeding
max.poll.interval.ms, causing rebalances.
Key points
- Key → partition → ordering
- Consumer groups scale up to #partitions
- Replication + acks=all for durability
Further reading: Apache Kafka: Design
Practice questions
A topic has 6 partitions; what happens if you run 10 consumers in one group?
Answer
Only 6 consumers receive partitions; 4 sit idle. Parallelism is limited by partition count.
What's a compacted topic used for?
Answer
Keeping the latest value per key indefinitely (e.g., current customer profile), useful for rebuilding state; older values for the same key are cleaned up.
Open in the app to mark lessons complete, take notes and track your progress.