Kafka Is Everywhere. So Why Do So Many Real-Time Systems Still Fall Short?

Apache Kafka has become the backbone of modern event streaming. Nearly every company doing anything serious with data at scale has Kafka somewhere in their stack. And that makes sense, it's reliable, battle-tested, and has a massive ecosystem around it.

But here's a pattern worth sitting with: the ubiquity of Kafka hasn't actually solved the hard problem of real-time reasoning. It's solved the transport problem. Those are not the same thing.

What Kafka Is Actually Good At

Kafka does one thing exceptionally well: durable, high-throughput message passing between systems. You can push millions of events per second through a well-tuned Kafka cluster and pull them out the other side with low latency and strong delivery guarantees.

For financial data, log aggregation, user activity streams, IoT telemetry, Kafka as a backbone makes a lot of sense. It decouples producers from consumers, absorbs spikes, and gives you a replay buffer when something downstream breaks.

That's genuinely useful infrastructure.

Where the Gap Opens Up

Kafka gets events from A to B. What happens at B is entirely up to you.

If B is a consumer that batches events, processes them in micro-windows, and emits aggregated results every few seconds, you've essentially rebuilt a slower pipeline with extra steps. You've added Kafka latency on top of batch latency, and called it streaming.

This is more common than you'd think. Teams wire up Kafka and assume they've "gone real-time." The broker is real-time. The processing logic often isn't.

The Statefulness Problem

The deeper issue is statefulness. Kafka itself is stateless in the sense that it doesn't understand the meaning of your event stream, it just stores and forwards. The moment you need to do something like:

• Detect that a price moved more than 2% in a rolling 30-second window
• Flag when a user's activity pattern deviates from their last 7-day baseline
• Alert when three correlated events occur within 500ms of each other

...you need stateful stream processing sitting on top. Something like Flink, ksqlDB, or a custom consumer that maintains in-memory state and handles late arrivals, out-of-order events, and watermarking.

Most tutorials skip this part. They show you how to produce and consume from a topic. They don't show you what happens when your join window is too narrow, or how to handle state explosion at scale, or what to do when your consumer group rebalances mid-computation.

The Latency Assumptions Are Often Wrong

Another underappreciated issue: Kafka's default configs are not tuned for ultra-low latency. linger.ms, batch size, replication acknowledgment settings, these all create latency headroom that batch pipelines don't care about but real-time applications do.

A naively configured Kafka cluster can add tens to hundreds of milliseconds to your end-to-end pipeline. For financial market data or live fraud detection, that's not acceptable. Getting Kafka to behave at true low latency requires deliberate tuning and usually trade-offs in throughput or durability.

What This Actually Means for Architecture

Kafka being the "de facto standard" for event streaming is both true and a little misleading. It's the de facto standard for the transport layer of streaming architectures. The processing, enrichment, state management, and real-time inference layers are still largely unsolved problems that teams rebuild from scratch on every project.

The real challenge in 2025 and beyond isn't getting events into a broker. It's doing something useful with those events at the speed they arrive, without buffering away the signal in the process.

That's the part worth spending more engineering time on.