CodeCrafters CodeCrafters

Build your own Kafka

Learn about TCP servers, the Kafka wire protocol and more.

Start Building
kafka
JavaScript
kafka
TypeScript
kafka
C++
kafka
Gleam
kafka
Zig
kafka
C#
kafka
Go
kafka
Java
kafka
Python
kafka
Rust
This challenge is free to try when it's in beta. We keep challenges in beta for a few weeks to gather feedback.

Apache Kafka is a distributed event streaming platform often used for high-performance data pipelines. In this challenge, you'll build your own Kafka broker that's capable of serving basic requests.

Along the way you'll learn about TCP servers, the Kafka wire protocol and more.

Stages

Bind to a port
Send Correlation ID
Parse Correlation ID
Parse API Version
Handle APIVersions requests

Concurrent Clients

Serial requests
Concurrent requests

Listing Partitions

Include DescribeTopicPartitions in APIVersions
List for an unknown topic
List for a single partition
List for multiple partitions
List for multiple topics

Consuming Messages

Include Fetch in APIVersions
Fetch with no topics
Fetch with an unknown topic
Fetch with an empty topic
Fetch single message from disk
Fetch multiple messages from disk
“ There are few sites I like as much that have a step by step guide. The real-time feedback is so good, it's creepy! avatar Ananthalakshmi Sankar Automation Engineer at Apple “ I think the instant feedback right there in the git push is really cool. Didn't even know that was possible! avatar Patrick Burris Senior Software Developer, CenturyLink