Duration: 14 Hours (2 Intensive Days) Focus: Hands-on Topic Lifecycle Management and Production Operations
Day 1: Topic Fundamentals and Lifecycle Management 🛠️
Module 1: Introduction to Kafka and Topic Architecture
- Overview of Apache Kafka (Architecture, producers, consumers, brokers, and its role in stream processing).
- Kafka Topic Fundamentals (Definition of a topic, partition concept, and offset management).
- Use Cases for Kafka topics (Message queues, stream processing, decoupled systems, event sourcing).
Module 2: Topic Creation and Management
- Creating Topics (Using Kafka command-line tools (
kafka-topics.sh) to create topics). - Topic Configuration (Setting crucial topic properties during creation: replication factor, number of partitions).
- Topic Modification (Using command-line tools to modify existing topic properties (e.g., adding partitions)).
- Topic Deletion (Safely deleting topics and understanding configuration flags).
Module 3: Security and Access Control
- Security Overview (Overview of Kafka security features).
- Secure Communication (Configuring SSL or TLS for secure communication between brokers, producers, and consumers).
- Authentication (Setting up authentication mechanisms (e.g., SASL)).
- Authorization (ACLs) (Configuring fine-grained authorization (Access Control Lists) for topics).
Module 4: High Availability and Disaster Recovery
- Replication for HA (Ensuring high availability with replication factor settings and understanding in-sync replicas (ISRs)).
- Disaster Recovery Strategies (Implementing cross-cluster replication (e.g., MirrorMaker) for Disaster recovery).
- Failover Handling (Monitoring and handling failover events and ensuring data recovery).
Day 2: Performance, Maintenance, and Advanced Topics 📈
Module 5: Monitoring Topic Health and Performance
- Monitoring Importance (Understanding the importance of monitoring Kafka topics).
- Key Metrics (Identifying critical metrics (latency, throughput, consumer lag) and available monitoring tools (e.g., JMX, Prometheus/Grafana)).
- Topic Health Check (Monitoring topic health and performance indicators).
Module 6: Scaling and Performance Tuning
- Scaling Strategies (Methods for scaling Kafka topics by adjusting partitions and cluster size).
- Performance Tuning Techniques (Optimizing broker and topic settings for low latency or high throughput).
- Partition Management (Adding partitions dynamically and understanding the impact on consumers).
Module 7: Data Retention and Cleanup
- Retention Policies (Configuring retention policies based on time and size (
log.retention.hours,log.retention.bytes)). - Disk Space Management (Strategies for managing disk space usage).
- Cleanup Methods (Data cleanup using deletion and log compaction (
log.cleanup.policy=compact)).
Module 8: Best Practices and Future Trends
- Topic Administration Best Practices (Recommended naming conventions, partition sizing, and monitoring setup).
- Advanced Management (Advanced topic management techniques (e.g., preferred leader election, reassigning partitions)).
- Use Case Considerations (Tailoring topic setup to use case-specific considerations (e.g., stream processing vs. messaging)).
- Future Outlook (Review of Future trends and developments in Kafka topic administration (e.g., Tiered Storage)).