Kafka Streams API for Developers Online Course

Kafka Streams API for Developers Online Course

This course takes you deep into Kafka Streams development, starting with fundamental concepts and building your first streaming application locally. You’ll explore core operators, serialization and deserialization techniques, and implement an order management system for a retail company. Key topics include error handling, KTable and GlobalKTable usage, stateful operators, aggregations, rekeying records, and joins. You’ll also learn automated testing with unit and integration tests using Embedded Kafka, understand grace periods, and package your app for deployment. By the end, you’ll have the skills to design and launch robust Kafka Streams applications.

Who should take this Course?

The Kafka Streams API for Developers Online Course is ideal for software developers, data engineers, and backend programmers who want to build real-time, event-driven applications using Apache Kafka’s Streams API. It is also suitable for students, architects, and professionals seeking hands-on experience in stream processing, stateful computations, and data pipeline development to manage and process high-volume data efficiently.

What you will learn

• Build advanced Kafka Streams applications using Streams API
• Build Kafka Streams application using high-level DSL
• Test Kafka Streams using TopologyTestDriver using JUnit5
• Test Spring Kafka Streams using EmbeddedKafka and JUnit5
• Aggregate multiple events into aggregated events
• Learn to join multiple streams into one joined stream

Course Outline

Getting Started with the Course

• Course Introduction
• Prerequisites

Getting Started with Kafka Streams

• Introduction to Kafka Streams
• Kafka Streams Terminologies - Topology and Processor
• Introduction to KStreams API

Greetings Kafka Streams App Using KStreams API

• Overview of the Greetings App
• Set Up the Greetings App
• Topology of the Greetings App
• Build the Kafka Streams Launcher Application
• Setting Up Kafka Environment and Testing Our Greeting App

Operators in Kafka Streams Using KStreams API

• Filter and FilterNot
• Map/MapValues
• FlatMapValues/FlatMap
• peek
• merge

Serialization and Deserialization in Kafka Streams

• How Key/Value Serialization and Deserialization Works in Kafka Streams
• Providing Default Serializer/Deserializer Using Application Configuration
• Build Custom Serdes for Enhanced Greeting Messages
• Usage of Custom Serde in the Greeting App

Reusable Generic Serializer/Deserializer (Recommended Approach)

• Build a Generic Serializer/Deserializer
• Integrate Generic Serializer/Deserializer into the Greeting App

Order Management Kafka Streams Application - A Real-Time Use Case

• Overview of the Retail App
• Build the Topology for the Orders Management App
• Split the Restaurant/Retail Shopping Orders - Using Split and Branch Operator
• Transform the Order Domain to a Revenue Domain Type

Topology, Stream, and Tasks - Under the Hood

• Internals of Topology, Stream, and Tasks
• Explore the Behavior of Streams by Modifying the Stream Threads

Error/Exception Handling in Kafka Streams

• Failures in Kafka Streams
• Default Deserialization Error Behavior
• Custom Deserialization Error Handler
• Default and Custom Processor Error Handler
• Custom Production Error Handler
• Error Handling When Kafka Cluster Is Down

KTable and Global KTable

• Introduction to KTable API
• Build a Topology for KTable
• KTable - Under the Hood
• GlobalKTable

StateFul Operations in Kafka Streams - Aggregate, Join, and Windowing Events

• StateFul Operations in Kafka Streams
• What Is Aggregation and How It Works?
• Aggregation Using "count" Operator
• Aggregation Using "reduce" Operator
• Aggregation Using "aggregate" Operator
• Using Materialized Store for count and reduce Operators

StateFul Operation Results - How to Access Them?

• How to Access the Results of Aggregation

Aggregation in Order Management Application - A Real-Time Use Case

• Total Number of Orders by Each Store Using "count" Operator
• Total Revenue by Each Store Using "aggregate" Operator

Rekeying Kafka Records for Stateful Operations

• Effect of null Key in Stateful Operations and Repartition of Kafka Records
• Rekeying Using the "selectKey" Operator

StateFul Operations in Kafka Streams - Join

• Introduction to Joins and Types of Joins in Kafka Streams
• Explore innerJoin Using "join" Operator - Joining KStream and KTable
• Explore innerJoin Using "join" Operator - Joining KStream and GlobalKTable
• Explore innerJoin Using "join" Operator - Joining KTable and KTable
• Explore innerJoin Using "join" Operator - Joining KStream and KStream
• Joining Kafka Streams Using "leftJoin" Operator
• Joining Kafka Streams Using "outerJoin" Operator
• Join - Under the Hood
• Co-Partitioning Requirements in Joins

Join in Order Management Application - A Real-Time Use Case

• Join Aggregate Revenue with StoreDetails KTable

StateFul Operations in Kafka Streams - Windowing

• Introduction to Windowing and Time Concepts
• Windowing in Kafka Streams - Tumbling Windows
• Control Emission of Windowed Results Using "suppress" Operator
• Windowing in Kafka Streams - Hopping Windows
• Windowing in Kafka Streams - Sliding Windows

Widowing in Order Management Application - A Real-Time Use Case

• New Requirements for the Order Management Application
• Implementing a CustomTimeStamp Extractor
• Aggregate "Number of Orders" by Windows
• Aggregate Revenue by Windows
• Joins on the Windowed Data

Behavior of Records with Future and Older Timestamp in Windowing

• Records with Timestamps before and after the CurrentTimestamp

Build Kafka Streams Application Using Spring Boot

• Introduction to Spring Boot and Kafka Streams
• Set Up the Project - Greeting Streams App Using Spring Kafka Streams
• Configuring the Kafka Stream Using application.yml
• Build the Greeting Topology
• Test Greeting App in Local

Spring Boot Autoconfiguration of Kafka Streams

• Internals of Autoconfiguring Kafka Streams in Spring Boot

JSON Serialization/Deserialization in Spring Kafka Streams

• JSON Serialization/Deserialization Using JsonSerde
• JsonSerde Using Custom ObjectMapper

Error Handling in Spring Kafka Streams

• Handle Deserialization Error - Approach 1
• Handle Deserialization Error Using Custom Error Handler - Approach 2
• Handle Deserialization Errors Using Spring Specific Approach- Approach 3
• Handle Uncaught Exceptions in the Topology
• Handle Production Errors

Build Orders Kafka Streams Application Using Spring Boot

• Set Up the Base Project for Orders Kafka Streams App
• Create the OrdersTopology

Interactive Queries - Querying State Stores Using RESTFUL APIs

• Build a GET Endpoint to Retrieve the OrderCount by OrderType - Part 1
• Build a GET Endpoint to Retrieve the OrderCount by OrderType - Part 2
• Retrieve OrderCount by OrderType and LocationId
• Build a GET Endpoint to Retrieve the OrderCount for All OrderTypes
• Build a GET Endpoint to Retrieve the Revenue by OrderType
• Global Error Handling for Useful Client Error Messages

Interactive Queries - Querying Window State Stores Using RESTFUL APIs

• Build a GET Endpoint to Retrieve OrderCount by OrderType
• Build a GET Endpoint to Retrieve the Windowed OrderCount for All OrderTypes
• Build a GET Endpoint to Retrieve the Windowed OrderCount within a Time Range
• Build a GET Endpoint to Retrieve the Revenue by OrderType

Testing Kafka Streams Using TopologyTestDriver and JUnit5

• Testing Kafka Streams Using TopologyTestDriver
• Unit Testing Greetings App - Writing Data to a Output Topic
• Unit Testing Greetings App - Testing Multiple Messages
• Unit Testing Greetings App - Error Scenario
• Unit Testing OrdersCount - Writing Data to a State Store
• Unit Testing OrdersRevenue - Writing Data to a State Store
• Unit Testing OrdersRevenue by Windows - Writing Data to a State Store
• Limitations of TopologyTestDriver

Testing Kafka Streams in Spring Boot Using TopologyTestDriver and JUnit5

• Unit Test Using TopologyTestDriver in Spring Boot

Integration Testing Spring KafkaStreams App Using @EmbeddedKafka

• Introduction and Set Up Integration Test
• Integration Test for OrdersCount
• Integration Test for OrdersRevenue
• Integration Test for OrdersRevenue By Windows

Grace Period in Kafka Streams

• Grace Period in Windowing

Build and Package the Spring Boot App as an Executable

• Package the Spring Boot App and Execute It as a Jar File

Exactly Once Processing/Semantics in Kafka Streams

• What Is Exactly Once Processing and Why Is It Needed?
• Set Up Exactly Once Processing in Kafka Streams
• Transactions and Idempotent Producer - Under the Hood
• Limitations and Performance Impacts of Kafka Transactions

Running Kafka Streams Applications as Multiple Instances (Spring Boot)

• Running Kafka Streams Applications as Multiple Instances
• Set Up to Run the Kafka Streams as Multiple Instances
• Kafka Streams Metadata
• Aggregate Data from Multiple Instances - Overview
• Aggregate Data from Multiple Instances - Fetching Metadata - Part 1
• Aggregate Data from Multiple Instances - Building RestClients - Part 2
• Aggregate Data from Multiple Instances - Testing End to End - Part 3
• Key-Based Queries with Multiple Instances - Overview
• Key-Based Queries Multiple Instances - Fetching Metadata- Part 1
• Key-Based Queries Multiple Instances - Building RestClient and Testing- Part 2
• Refactor the Code for OrderCount for All OrderTypes Endpoint - /v1/orders/count
• Fix the Test Cases
• What about the Other Endpoints?