{"id":161,"date":"2026-03-09T00:43:21","date_gmt":"2026-03-09T00:43:21","guid":{"rendered":"https:\/\/blog.rebalai.com\/en\/2026\/03\/09\/event-driven-architecture-in-2026-why-microservice\/"},"modified":"2026-03-18T22:00:07","modified_gmt":"2026-03-18T22:00:07","slug":"event-driven-architecture-in-2026-why-microservice","status":"publish","type":"post","link":"https:\/\/blog.rebalai.com\/en\/2026\/03\/09\/event-driven-architecture-in-2026-why-microservice\/","title":{"rendered":"Event-Driven Architecture in 2026: Why My Microservices Finally Stopped Talking Back"},"content":{"rendered":"

At 2:14am on a Tuesday, my phone buzzed. The order service was timing out. Which was timing out the inventory service. Which was failing to respond to the notification service. Which caused the analytics pipeline to back up and start throwing 503s. Six microservices<\/a>, six separate REST calls in a chain, and the entire thing had collapsed like a card tower because one database query ran long.<\/p>\n

I fixed it by 3am, pushed a hotfix, and then sat there thinking: there has to be a better way to structure this.<\/p>\n

That was about 18 months ago. I’ve since migrated two projects \u2014 a small SaaS I co-own and a client engagement for a 12-person startup \u2014 to event-driven architectures built on top of Redpanda and Kafka respectively. Here’s what I actually learned<\/a>, including the embarrassing parts.<\/p>\n

The Core Problem With REST-Coupled Microservices<\/h2>\n

Look, REST makes sense for<\/a> request-response. Client needs data, server provides it. Clean. Simple. But when you’re building microservices<\/a> that need to react to things that happened<\/em>, REST starts to fight you.<\/p>\n

The pattern I kept falling into was something like: Service A does a thing, then synchronously calls Service B, which calls Service C, which maybe calls Service D. You end up with temporal coupling \u2014 every service in the<\/a> chain needs to be healthy right now<\/em> for any individual operation to succeed. And that 2am incident wasn’t a fluke. I counted seven similar cascades in for Production<\/a> Workloads” rel=”nofollow sponsored” target=”_blank”>production<\/a> logs from the three months<\/a> before I finally migrated.<\/p>\n

Here is the thing: the services weren’t doing wrong things. The architecture<\/em> was wrong for what we needed. Order placed? That’s an event. Something happened. The inventory service doesn’t need to be called synchronously \u2014 it needs to know<\/em> that the order happened and update its state accordingly. Same with notifications, same with analytics.<\/p>\n

Event streaming flips the dependency direction. The order service doesn’t know (or care) who’s listening. It publishes an order.created<\/code> event to a topic and moves on. Inventory, notifications, analytics \u2014 all consumers, all independent, all replayable if something goes wrong.<\/p>\n

The other thing that took me embarrassingly long to appreciate: replay. With REST, if your analytics service was down for two hours, those events are gone. With a stream, they’re retained. You bring the service back up, it consumes from its last committed offset, and catches up. I pushed this on a Friday afternoon the first time and spent about 20 minutes terrified watching the consumer lag spike \u2014 but it caught up in<\/a> under four minutes on roughly 40k missed events.<\/p>\n

Choosing a Broker: Kafka, Redpanda, and NATS (My Honest Take)<\/h2>\n

I’ve used all three in<\/a> production<\/a> now. They’re not interchangeable despite what some blog posts suggest.<\/p>\n

Kafka<\/strong> is the industry standard for a reason. If you’re running on a managed service like Confluent Cloud or AWS<\/a> MSK, operationally it’s fine \u2014 you’re not managing ZooKeeper (or KRaft, which became the default with Kafka 3.3) yourself. The ecosystem is enormous. The Kafka Connect ecosystem alone saved me probably a week of work on one client project where we needed to sync events into Postgres. That said, for a team of two or three, even managed Kafka felt overweight for our<\/a> simpler SaaS needs.<\/p>\n

Redpanda<\/strong> is where I landed for the SaaS project, and honestly I’m glad I did. It’s Kafka-compatible \u2014 you use the same client libraries, same topic\/partition\/consumer group model \u2014 but it’s a single binary with no JVM, no ZooKeeper, and meaningfully lower tail latencies on p99. The development experience is better too: docker run -p 9092:9092 redpandadata\/redpanda<\/code> and you have a real broker running locally in about four seconds. Not a mock, not an in-memory fake \u2014 the actual thing.<\/p>\n

NATS JetStream<\/strong> is the outlier. Lower operational overhead, simpler mental model, genuinely impressive throughput for its resource footprint. But the consumer semantics are different enough from Kafka that your engineers need to relearn some things, and the ecosystem is smaller. I used it for a side project and liked it. I probably wouldn’t reach for it on a team that already knows Kafka concepts.<\/p>\n

One thing I noticed: if you’re already deep in Kubernetes<\/a> and have a platform team, Kafka (managed) is the obvious choice. If you’re a small team self-hosting, Redpanda is worth serious consideration. I am not 100% sure Redpanda holds up<\/a> at Confluent-scale workloads \u2014 I haven’t stress-tested it there \u2014 but for teams under 20 engineers, it’s been solid.<\/p>\n

Schema Registry and the Mistake That Bit Me<\/h2>\n

Right, so \u2014 this is the part I wish someone had told me before I started.<\/p>\n

When you move to event streaming, your event schema becomes a contract between services. With REST, you control the API and its versioning. With events, consumers are decoupled and may be running different versions at the<\/a> same time. If you just publish raw JSON and one day add a required field, you will break older consumers silently. Ask me how I know.<\/p>\n

I pushed a change to the order.created<\/code> event on a Wednesday, added a shipping_priority<\/code> field that the fulfillment service expected to always be present. The notification service \u2014 which I had forgotten<\/em> was also consuming that topic \u2014 started throwing null pointer exceptions about six hours later when someone placed an order. It wasn’t obvious, it wasn’t loud, it just quietly started failing deserialization.<\/p>\n

The fix is a schema registry. Confluent ships one, Redpanda has one built in, and Apicurio is a good open source option if you want to run your own. You register your schemas (Avro and Protobuf are both well supported; I prefer Protobuf for the generated types), and producers\/consumers validate against them at runtime. More importantly, you can enforce compatibility rules \u2014 BACKWARD<\/code> compatibility means new schemas can read old messages, FULL<\/code> compatibility means both directions \u2014 and the registry will reject breaking changes before they reach production<\/a>.<\/p>\n

Here’s a simplified producer setup showing schema validation in Python<\/a> using Confluent’s client:<\/p>\n

from confluent_kafka import Producer\nfrom confluent_kafka.schema_registry import SchemaRegistryClient\nfrom confluent_kafka.schema_registry.protobuf import ProtobufSerializer\nfrom confluent_kafka.serialization import SerializationContext, MessageField\n\nfrom events_pb2 import OrderCreatedEvent  # generated from .proto\n\nschema_registry = SchemaRegistryClient({"url": "http:\/\/localhost:8081"})\nserializer = ProtobufSerializer(\n    OrderCreatedEvent,\n    schema_registry,\n    {"use.deprecated.format": False}\n)\n\nproducer = Producer({"bootstrap.servers": "localhost:9092"})\n\ndef publish_order_created(order_id: str, customer_id: str, total_cents: int):\n    event = OrderCreatedEvent(\n        order_id=order_id,\n        customer_id=customer_id,\n        total_cents=total_cents,\n        # shipping_priority added later \u2014 default value handles old consumers\n    )\n    producer.produce(\n        topic="order.created",\n        value=serializer(event, SerializationContext("order.created", MessageField.VALUE)),\n        key=order_id,\n    )\n    producer.flush()\n<\/code><\/pre>\n
The consumer side mirrors this with a ProtobufDeserializer<\/code>. The key point: if you try to register a schema that breaks BACKWARD<\/code> compatibility, the registry rejects it. You catch the problem at deploy<\/a> time, not six hours after the fact in production<\/a>.<\/p>\n
Choreography vs Orchestration \u2014 And When I Got It Completely Wrong<\/h2>\n
I thought choreography was always better. Services react to events independently, no central coordinator, maximum decoupling \u2014 what’s not to like?<\/p>\n
Choreography works beautifully for simple flows. order.created<\/code> fires, inventory decrements, notification sends, analytics records. Each service does its thing. No one’s in charge. Very elegant.<\/p>\n
But I tried to apply it to a checkout flow that had conditional branches: if payment failed, roll back inventory reservation; if the customer was on a trial plan, skip certain fulfillment steps; if the address flagged a fraud check, pause the whole thing pending review. Four services, three conditional branches, two possible rollback paths.<\/p>\n
Six weeks in<\/a>, I had an event topology I could no longer reason about. Which service emitted which events, under which conditions, with what side effects \u2014 it was all implicit, scattered across codebases, and nearly impossible to trace when something went wrong. One of my teammates described it as<\/a> “the distributed state machine that hides in the<\/a> dark.”<\/p>\n
For complex, stateful workflows with branching logic and rollback requirements, orchestration is better. Tools like Temporal.io (which I’ve been using for about eight months now) or Apache Flink for stream processing give you a place where the business logic lives<\/em> \u2014 one service that explicitly manages the workflow state. The other services are still event-driven at the<\/a> edges; they still emit and consume events. But there’s a coordinator that knows where in the<\/a> process you are.<\/p>\n
I’m not saying choreography is wrong. I applied it past the point where it<\/a> was the right tool \u2014 that’s on me. My heuristic now: if you need to visualize the workflow on a whiteboard with more than two decision branches, you want orchestration for that flow.<\/p>\n
What Actually Changed<\/a> After the Migration<\/h2>\n
The 2am pages stopped. That’s the simplest way to put it. Not entirely \u2014 distributed systems are<\/a> still distributed systems \u2014 but the cascading timeouts that characterized our worst incidents went away because the dependency chains are gone. The order service can run and publish events even if the notification service is deploying a new version.<\/p>\n
Consumer lag monitoring became the new on-call skill. Where before I watched error rates and latency on individual endpoints, now I also watch whether consumers are keeping up with their topics. Grafana + Prometheus scraping Kafka\/Redpanda metrics for consumer_lag_sum<\/code> is how I set that up, and the dashboard has been more predictive than anything I had before.<\/p>\n
Debugging changed character. The good<\/em> news: every event is persisted on the topic, so you can replay and inspect exactly what happened. I’ve used this multiple times to reconstruct bugs that would have been untraceable with ephemeral REST calls. The challenging<\/em> news: distributed tracing across async boundaries is harder to wire up properly. I’m using OpenTelemetry with trace context propagated in event headers, which works, but it took<\/a> a weekend to get right and the documentation is still a bit fragmented.<\/p>\n
Idempotency is not optional. Consumers can receive the same event more than once (at-least-once delivery is the default in Kafka and Redpanda). Every consumer I wrote now checks for duplicate processing using the event ID. Early on I missed this in the<\/a> inventory service and briefly had orders double-decrementing stock. Thankfully this was caught in staging.<\/p>\n
What I Would Actually Recommend<\/h2>\n
If you’re running microservices<\/a> that communicate primarily through synchronous REST chains \u2014 especially if you’ve seen cascade failures or you’re bolting on an increasing number of webhooks and polling jobs \u2014 event streaming is worth the migration cost. The learning curve is real and so is the operational overhead, but so is the alternative: 2am pages that don’t have to happen.<\/p>\n
For most small-to-mid-sized teams, I’d start with Redpanda (self-hosted) or MSK (if you’re AWS<\/a>-native and want managed). Set up a schema registry from day one \u2014 not later, not “when we need it.” Use Protobuf or Avro for your schemas, not JSON. Pick BACKWARD<\/code> compatibility as your default and be explicit about breaking changes.<\/p>\n
Start by identifying one flow in your system that has multiple downstream dependents and convert just that flow to events. Don’t try to migrate everything at once. The first event topic will teach you more about your system’s actual shape than any architecture diagram ever did.<\/p>\n
For simple reactive flows, choreography is fine. Once you’ve got conditional branches, rollbacks, or anything that needs to be visualized like a flowchart \u2014 reach for Temporal or an equivalent. Getting that distinction right early saves the six weeks of<\/a> untangling I had to do.<\/p>\n
And monitor consumer lag like you monitor error rates. That number will tell you<\/a> when something’s wrong before your users do.<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
At 2:14am on a Tuesday, my phone buzzed. The order service was timing out. Which was timing out the inventory service.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[1],"tags":[],"class_list":["post-161","post","type-post","status-publish","format-standard","hentry","category-general"],"_links":{"self":[{"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/posts\/161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/comments?post=161"}],"version-history":[{"count":20,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/posts\/161\/revisions"}],"predecessor-version":[{"id":510,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/posts\/161\/revisions\/510"}],"wp:attachment":[{"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/media?parent=161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/categories?post=161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.rebalai.com\/en\/wp-json\/wp\/v2\/tags?post=161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}