Overview
Apache Kafka is a distributed streaming platform. It is de facto a standard for building data pipelines and it solves a lot of different use-cases around data processing: it can be used as a message queue, distributed log, stream processor, etc.
We’ll start with some theory behind data pipelines in general, then continue with fundamental concepts behind Kafka. We’ll also discover important components like Kafka Streams and Kafka Connect.
Requirements
Basic Java 8 or Scala knowledge is preferable. Please install Docker and Docker Compose if you want to run examples locally.
Course Outline
- Data pipelines 101: ingestion, storage, processing
- Kafka fundamentals: topics, partitions, brokers, replication, etc.
- Producer and Consumer APIs
- Kafka Streams as a processing layer
- Kafka Connect for integrating with external systems
- Kafka best practices and tuning