|
Modern applications rarely do just one thing at a time. An API request creates an order, and then another service needs to reserve stock, another to charge the customer, another to send an email, and so on. In a serverless or event-driven architecture, follow-up actions are usually triggered by messages (either events or commands). That gives us loose coupling, better scalability, and independent services. But it also introduces a reliability problem. “What happens when the database update succeeds, but publishing the event fails?” Or the opposite: “What happens when a consumer processes an event successfully, but crashes before acknowledging it?” This is where the Outbox Pattern and Inbox Pattern come in. The TL;DR is:
They are often used together to build reliable event processing flows. The problem with naive event publishingImagine an The simplest implementation might look like this: This looks fine, but it has a failure case. Now, the order exists, but the event was never published. Downstream services do not know the order exists. Inventory is not reserved. Payment is not started. The customer does not receive a confirmation email. The mistake here is treating the database write and the event publish as if they are one atomic operation. They are not. This is a distributed transaction problem. The Outbox PatternThe Outbox pattern solves the “database updated but event not published” problem. Instead of writing to the database and then immediately publishing to EventBridge, the service writes both the order and the outgoing event (the outbox item) in a single database transaction. Later on, another process will pick up the outbox item and publish it to EventBridge asynchronously. The idea is simple: first persist the intent to publish, then publish later. You can easily implement the Outbox pattern with DynamoDB and EventBridge Pipes. For example, you’d have an Orders table and an Outbox table and use From here, DynamoDB Streams captures the A raw DynamoDB stream record is a database change notification. It contains DynamoDB-specific structure, such as That is not what you want downstream services to consume. Instead, the pipe should transform the DynamoDB There are many variations to this basic pattern. For example, you may replace EventBridge Pipes with a Lambda function if the publishing step has real business logic. Or you may forego the Outbox table altogether and derive the outgoing event from the inserted Order item. However, if you wish to track the status of the outbox item (e.g. from The Inbox PatternThe Inbox pattern solves the duplicate delivery problem on the consumer side. Most event and messaging systems should be treated as at least once delivery systems from the application’s point of view. Even if the infrastructure reduces duplicates, your business logic should still assume duplicates are possible. This is especially true for Lambda, because all async invocations (e.g. via EventBridge or SNS) have at least-once delivery. Without protection, the payment service might charge the customer twice! The Inbox Pattern prevents that by recording which messages have already been processed. The consumer stores the incoming event ID in an Inbox table, and the inbox item then acts as a deduplication key. The key to implementing this pattern with DynamoDB is to use conditional writes to ensure the event ID does not exist in the Inbox table already. Inbox vs. IdempotencyYou might be wondering, “How is this different from using simple idempotency keys?” These are related ideas that mainly differ in scope. An idempotency key protects one business operation. Whereas the Inbox pattern protects message processing. Within one message processing, there might be several business operations, each with its own idempotency key. The Inbox pattern is idempotency applied at the message processing boundary, using the message/event ID as the deduplication key. Inbox item first, or business update first?Avoid this flow:
If step 1 succeeds and step 3 fails, future retries will see the inbox item and skip processing, even though the business action never happened. Also, avoid this flow:
If step 1 succeeds and step 3 fails, future retries may repeat the business action. The safest solution is to write the inbox item and business change in the same transaction. But what if that’s not possible? For example, what if there are intermediate side-effecting actions, such as calling a third-party API? In that case, this is a safer flow:
If the first worker fails or times out, a subsequent retry can resume processing after the lock has expired. But what about the side effects that the first worker has already performed? That’s where we also need idempotency keys to protect the individual side-effecting actions! Final thoughtsMost real-world event-driven systems have both consumers and producers. For example, the payment service consumes That means it needs both patterns:
These patterns are not about making failures impossible. They are about making failures safe. The Outbox pattern accepts that publishing can fail, so it first stores the intent to publish. The Inbox pattern accepts that duplicate deliveries can happen, so it makes message handling idempotent. Together, they give you a strong foundation for reliable event processing. Related articles
|
Join 17K readers and level up you AWS game with just 5 mins a week.
If you use Claude Code a lot, you’ve probably run into usage limits, sometimes even in short coding sessions. But cost isn’t the only problem. In long-running sessions, the context window eventually fills up, and that can cause the agent to forget earlier decisions, lose important details, or come back from compaction with gaps in its working memory. Here are three tools worth checking out if you want to reduce token usage and make longer coding sessions possible. 1. CavemanThis is a Claude...
AI agents can now scan an entire open-source codebase for exploitable vulnerabilities in hours. Frontier models carry the complete library of known bug classes in their weights. So you can simply point an AI agent at a codebase and tell it to find zero-days. This isn't theoretical. Willy Tarreau, the HAProxy lead developer, reports that security bug reports have jumped from 2–3 per week to 5–10 per day. Greg Kroah-Hartman, the Linux kernel maintainer, described what happened: "Months ago, we...
Lambda Durable Functions makes it easy to implement business workflows using plain Lambda functions. Besides the intended use cases, they also let us implement ETL jobs without needing recursions or Step Functions. Many long-running ETL jobs have a time-consuming, sequential steps that cannot be easily parallelised. For example: Fetching data from shared databases/APIs with throughput limits. When data needs to be processed sequentially. Historically, Lambda was not a good fit for these...