Duration: 14 Hours (2 Intensive Days) Focus: Configuring Kafka for Reliable Transactional and Asynchronous Queuing
Day 1: Foundational Concepts and Message Flow 📩
Module 1: Introduction to Kafka for Message Queuing
- Overview of Apache Kafka (Review of core architecture: brokers, producers, consumers, and ZooKeeper/KRaft).
- Queuing Concepts (Kafka as a message queue: key concepts and characteristics—high throughput, durable storage, ordered partitions).
- Use Cases (Identifying common use cases for using Kafka as a message queue, such as decoupled microservices and asynchronous processing).
Module 2: Topic Configuration for Queuing Scenarios
- Topic Configuration (Topic configuration for message queuing scenarios—e.g., using a high replication factor, setting appropriate retention).
- Partitioning Strategies (Partitioning strategies for message distribution to maximize parallel consumption and maintain order within partitions).
- Fault Tolerance (Replication and fault tolerance considerations—understanding In-Sync Replicas (ISRs) and rack awareness).
Module 3: Message Production and Consumption
- Producing Messages (Implementing producers, understanding acknowledgements (acks), and configuring for reliable message delivery).
- Implementing Consumers (Developing message consumers and setting isolation levels).
- Consumer Group Management (Managing consumer groups and offsets for message consumption, including balancing partitions across instances).
Day 2: Reliability, Monitoring, and Security 🔒
Module 4: Message Processing Guarantees and Error Handling
- Processing Semantics (Understanding message processing semantics in Kafka: At-Most-Once, At-Least-Once, and Exactly-Once delivery).
- Error Handling (Error handling and retry strategies within consumer applications).
- Message Reprocessing (Implementing Dead-letter queues and message reprocessing patterns for failed messages).
Module 5: Monitoring and Management
- Throughput and Latency (Monitoring Kafka for message throughput and latency metrics).
- Management Tools (Managing Kafka topics and consumer groups using command-line tools and web UIs).
- Performance Tuning (Performance tuning for message queuing workloads: batching, compression, and adjusting thread pools).
Module 6: Security and Compliance
- Security Measures (Implementing comprehensive security measures for Kafka message queuing environments).
- Authentication and Authorization (Configuring Authentication and authorization (ACLs) for producers and consumers).
- Compliance Considerations (Compliance considerations for message storage and processing, including data encryption and retention policies).