The New Wave of the Streaming Log Technologies

When Kafka compatibility is no longer a goal.

Apache Kafka was open-sourced in 2011, and it made a huge impact a few years after that. The Streaming log became a very popular abstraction.

Today, 15 years later, I believe we’re experiencing the third wave of streaming log technologies. The abstraction is still the same (and as useful as before), but many new implementations raise and try to answer really hard questions.

A Bit of History

Was Apache Kafka actually in the first wave? Probably.

Before Kafka, the idea of an Enterprise Message Bus was quite common. We had messaging tools (IBM MQ, RabbitMQ, ActiveMQ, ZeroMQ, TIBCO, Solace and many more) of all shapes and flavours. Messaging and streaming share many primitives and concepts. And I do believe we needed to explore what was possible with messaging before arriving at the streaming log idea.

But Kafka had a unique twist: the log is durable, the fanout reads are cheap (you can have hundreds or thousands of consumers), and write performance is critical. Making it open-source contributed to its success as well.

After a while, many companies started offering a packaged Kafka product. Sometimes with a lot of customizations. I’d still categorize these products (Confluent, Aiven, AWS MSK, etc.) as the first wave: in the end, they made Kafka operationally easy, but they haven’t drastically changed the technology itself.

The second wave was started by Redpanda. It showed that it’s possible to re-implement such a complex project from scratch and maintain compatibility. Later came WarpStream and Bufstream, building on top of the object storage abstraction. Apache Pulsar is worth mentioning as well - it wasn’t created with Apache Kafka support, but it was eventually added.

The Third Wave

Apache Iggy

Apache Iggy is a high-performance streaming platform. It supports many primitives that can be found in Apache Kafka: topics, partitions, consumers, producers, etc. However, it doesn’t support the Kafka protocol, and it’s quite intentional. This fact alone made me very curious about the project.

Performance is the main goal of the project. For example, it uses Rust, thread-per-core architecture, io_uring, etc. This makes it somewhat similar to Redpanda (minus the Rust part).

It comes with a few distinct features:

• It supports more than one protocol: not just TCP, but also QUIC, HTTP and WebSocket (!). Supporting the last two means that it’s (theoretically) possible to consume or produce data directly in your browser (without any intermediaries).
• A very easy way to build and load connectors: just compile a dynamic library and load it at runtime on the broker side.

S2

S2 is a low-level API for real-time streams. It tries to answer the question: can interacting with a streaming log be as simple as reading and writing data with S3? There are no consumers and producers. There are no partitions. Instead, you have a few simple primitives:

• Streams (similar to topics).
• Appending to a stream.
• Reading a stream.

That’s it. Interacting with streams can be done over HTTP.

Durability is an important guarantee S2 offers. S2 solved our problem with reliably streaming long-running AI sessions. Before, connection drops could cause lost data and broken streams. S2's bottomless storage means our customers can stream for hours and network hiccups don’t matter.

This means that the simple low-level interface allows S2 to stream literally anything, not just conventional events/messages.

OpenData Log

OpenData is a family of open-source, object-native databases, and the Log is the latest addition.

The announcement blog post argues that the streaming logs are supposed to support two different types of workloads:

• Funnelling: collecting data from high-cardinality sources and coalescing them onto a low-cardinality pipe.
• Routing: delivering events from addressable sources to addressable destinations.

And Apache Kafka failed to properly support routing workloads. The post explains that using partitions for routing leads to large read amplification: e.g., you need to read the whole topic to find messages with certain keys.

OpenData Log also supports HTTP protocol with several endpoints for reading and writing records.

Honorable Mentions

• Apache Fluss, lakehouse-native streaming storage. Imagine a Kafka cluster, but every topic is a table with a defined schema. Arguably, it’s an example of a streaming log, but it’s specialized to serve data analytical use cases.
• Apache Pulsar 5. The next major version of Apache Pulsar will support Scalable Topics. A Scalable Topic is a partition-less topic that dynamically splits and merges its keyspace.

Summary

The new wave of streaming log technologies doesn’t support the Kafka protocol. Instead, they try to approach the log design differently.

In retrospect, S2 and OpenData Log are very similar: simple, low-level API endpoints over HTTP, Object Storage as a primary backend. OpenData Log decided to bet on routing use cases and designed the API around record keys.

I believe simplicity is the main goal behind the projects. Offloading a lot of complexity to object storage makes sense; you can cheaply get high durability and scalability guarantees.

Iggy stays very close to the Kafka protocol, but it’s not constrained by it. Having performance as the number one goal can be attractive for many potential users. And rewriting all your consumers and producers could be an easier sell nowadays, thanks to LLMs.

I still wonder if this new wave of systems will complement existing Kafka deployments, or try to replace them. Kafka is no longer a single project - it’s a protocol, and protocols are tenacious.