Transactional Outbox: Keeping Local Changes and Follow-Up Work Together

The problem with two separate writes

Suppose an order service needs to save an order in its database and notify a fulfillment service. These are two writes to separate systems, often called a dual-write problem.

Changing their order does not remove the failure gap:

• Save the order, then send the message: the database commit may succeed, but the application can crash before sending. The order exists without the expected notification.
• Send the message, then save the order: the recipient may receive the notification even if the database transaction later rolls back. Fulfillment could begin for an order that was never committed.

Putting the send call inside a database transaction does not automatically solve this. Unless the external system participates in a suitable coordinated transaction, rolling back the database cannot retract a message it has already accepted.

The outbox changes what the application commits. Rather than trying to complete the external communication during the local transaction, it commits a durable record of the communication that needs to happen.

How a transactional outbox works

A typical implementation uses a business table, an outbox table, and a delivery process often called a relay.

1. The application begins a database transaction.
2. It writes the business change, such as a new order.
3. It writes an outbox record describing the corresponding message.
4. It commits the transaction.
5. The relay discovers the committed outbox record and attempts delivery.
6. After receiving the required delivery acknowledgment, the relay records progress according to its delivery protocol.

The local write can be represented with this illustrative pseudocode:

BEGIN TRANSACTION

  INSERT order (
    order_id,
    customer_id,
    status
  )

  INSERT outbox_message (
    message_id,
    event_type,
    order_id,
    payload
  )

COMMIT

The important property is not the table names. Both writes must participate in the same atomic transaction. Writing to two databases independently, even from the same application function, does not establish that guarantee.

If the transaction rolls back, neither record persists. If it commits, the order and the pending message both persist. The relay must observe committed records, not messages from transactions that might still roll back.

This does not make downstream work immediate. There is a period during which the order exists but fulfillment has not yet received or processed its notification. The application needs to represent that delay honestly.

What an outbox record needs to preserve

An outbox record should give the delivery system enough information to route, identify, and interpret the message without guessing what the original transaction intended.

Depending on the application, useful fields include:

• A stable message identifier: retained across retries so repeated deliveries can be recognized.
• The event or command type: what happened, or what another component is being asked to do.
• A business-entity identifier: such as the order identifier, distinct from the message identifier.
• The payload and its schema version: the information the recipient needs and the format it should expect.
• Routing information: the intended destination, topic, or routing key.
• Sequence information: where processing order matters.
• Operational metadata: creation time, attempt count, next eligible attempt, or delivery state.

Not every implementation stores all delivery metadata in the outbox row. Some relays track progress through database-log positions or separate records.

The payload also needs a deliberate meaning. If a message describes the order as it existed when the transaction committed, reconstructing it later from mutable order data may produce a different message. Store the required snapshot or provide a reliable reference to the relevant version.

Include only necessary data. Outbox records, broker messages, and diagnostic logs can create additional copies of sensitive information, each with its own access and retention requirements.

Delivering committed messages

Two common relay approaches are polling and change data capture.

Polling the outbox

A polling relay periodically queries for eligible records. Multiple workers need a coordination mechanism so they do not all treat the same record as exclusively theirs.

Implementations may use database locking, atomic claims, or time-limited leases. A claim needs a recovery path: if a worker crashes, another worker must eventually be able to continue the work.

Long network calls while holding database locks can create contention. Designs often separate a short claiming transaction from the delivery attempt, while accepting that expired claims or uncertain outcomes may lead to repeated delivery.

Reading committed database changes

A change data capture relay reads committed outbox changes through the database’s change stream or transaction log. This can reduce repeated polling, but it introduces operational responsibilities around connector health, saved positions, log retention, and recovery.

Neither approach removes the need for retry handling or recipient-side duplicate protection.

Defining what acknowledgment means

A broker acknowledgment may establish that the broker accepted a message under its configured durability guarantees. It usually does not establish that a downstream service completed the business action.

If the workflow needs evidence of fulfillment completion, that requires a separate result or acknowledgment at the business level. “Published to the broker” and “order fulfilled” are different states.

Why duplicate delivery remains possible

Consider this sequence:

1. The relay sends a message.
2. The broker accepts it.
3. The relay crashes before recording successful delivery.
4. After recovery, the relay sends the message again.

The second send is understandable: the relay lacks durable confirmation of the first attempt. Marking the message delivered before sending would introduce the opposite risk—losing the message if the relay crashed before delivery.

Transactional outbox implementations therefore commonly support at-least-once delivery: repeated delivery is possible. Eventual delivery still depends on functioning infrastructure, recovery, and an operational policy that does not silently abandon the work.

This is why idempotency remains important at the recipient.

Recipient deduplication and the inbox pattern

A recipient can record processed message identifiers and reject repeated processing. In an inbox pattern, the recipient can commit that record and its own business-state change in one local transaction.

For example, a fulfillment service might atomically record that message message-842 was handled and create the corresponding fulfillment task. A uniqueness constraint and appropriate transaction handling can coordinate concurrent deliveries.

Merely checking a message identifier in memory, or recording it separately from the business update, leaves failure gaps.

The boundary also moves rather than disappears. If the recipient must call another external service, its local inbox transaction does not make that remote effect atomic. That next step needs its own retry-safety and reconciliation design.

Retry safety depends on the complete chain of effects, not just the sender’s outbox.

Preserving order where it matters

An outbox does not automatically preserve every order that an application might need. Concurrent transactions, multiple relay workers, broker partitions, and recipient concurrency can all affect observed processing order.

Start by identifying the actual requirement. An order’s creation event may need to precede its cancellation event, while events for unrelated orders can be processed independently.

Supporting that requirement may involve:

• Assigning a sequence number within each business entity or stream.
• Coordinating concurrent updates when assigning those numbers.
• Routing related messages through the same ordered delivery path.
• Preventing relay concurrency from overtaking earlier messages.
• Having recipients detect gaps, duplicates, and out-of-order arrivals.

A creation timestamp alone is generally insufficient to establish a dependable business sequence. Likewise, ordered publication does not guarantee ordered completion if recipient workers process messages concurrently.

Ordering policy must also address failures. If one message repeatedly fails, should later messages for the same entity wait, or can they proceed safely? That is a business and recovery decision, not merely a queue setting.

Monitoring, failures, and cleanup

A durable pending record is useful only if someone or something continues attending to it. An outbox can preserve work while its relay remains broken indefinitely.

Useful operational signals include:

• The age of the oldest undelivered message.
• The size and growth rate of the pending backlog.
• Delivery latency and repeated failure rates.
• Expired worker claims or stalled change-stream progress.
• Messages held for investigation and their resolution status.

Retry policies should distinguish transient failures from conditions that need intervention. Backoff can reduce pressure during an outage; it cannot repair an invalid payload or missing authorization.

Moving a message to a dead-letter destination can make investigation manageable, but it is not successful delivery. Define who reviews held messages, what evidence they receive, and how corrected work resumes. These are practical hold and escalation responsibilities.

Cleanup should follow an explicit retention policy. Deleting acknowledged outbox records does not establish that every consumer has finished. Retention must account for audit needs, replay requirements, and the meaning of the acknowledgment.

Recipient deduplication records need a compatible retention policy too. If an old message can be replayed after its deduplication record has been deleted, the recipient may process it again.

When the pattern is useful

A transactional outbox is useful when a committed local change must reliably lead to asynchronous communication, and the application cannot safely lose that follow-up intent.

Common examples include recording an order and requesting fulfillment, saving an account change and publishing an event, or committing a document revision and scheduling downstream indexing.

It is not necessary for every background task. If all required changes fit within one database transaction, an outbox may add complexity without solving a separate problem. If losing a nonessential telemetry event is acceptable, a less durable approach may be proportionate.

A database-backed job queue can also fulfill the same local atomicity requirement when the job record is committed in the same transaction as the business change. The important question is the guarantee, not whether the implementation uses the word “outbox.”

The pattern also does not replace authorization. A committed delivery intent records what the workflow planned to do. If execution depends on conditions that may change, the system may need an execution-time check rather than treating the stored message as permanent permission.

A practical outbox review

Before relying on an outbox, review the complete path:

1. Atomicity: do the business change and pending message commit in the same transaction?
2. Message meaning: does the record preserve the intended payload, identity, and version?
3. Relay recovery: can delivery resume after a crash, expired claim, or lost acknowledgment?
4. Duplicate handling: can recipients safely handle repeated and concurrent deliveries?
5. Ordering: are necessary sequence guarantees maintained through processing, not just publication?
6. Visibility: can operators distinguish pending work, acknowledged delivery, completed business work, and unresolved outcomes?
7. Retention: do cleanup and replay policies preserve the required recovery and duplicate-protection windows?

The transactional outbox provides a specific foundation: a local change does not commit without also recording its required follow-up message. Reliable completion beyond that boundary depends on delivery, recipient behavior, observability, and recovery working together.