Re-implementing the Outbox Pattern in MongoDB

I have blogged previously about how we implemented the Outbox pattern on top of SQL Server and Websphere MQ to avoid distributed transactions/two-phase commits in an Event Sourcing microservice here. In that post, I explained that we use one transaction to persist the event and record an entry in an “Outbox” table to publish the event.

I now have to implement the Outbox pattern again to solve the exact problem but this time on top of different technologies, namely: Solace and MongoDB. Solace is a message broker similar to Websphere MQ, and publishing to it does not require any changes to our previous implementation. However, MongoDB poses an issue as its transactions only support writing to a single document. So how can we ensure that writing the event and creating the outbox entry happens atomically?

The answer is: we cannot! Not while they are in separate documents, anyway.

Therefore, we need to store the outbox information in the same document alongside the event. All our events are, therefore, wrapped in a class that holds the event payload alongside the outbox information. It is the only way we can guarantee atomicity.

The domain is oblivious to the existence of the outbox information as this is not a domain concern but rather an infrastructure one. Hence, the repository is decorated with an Outbox repository that attaches this information before persisting the event.

Just like in the Outbox pattern I previously implemented, Hangfire is used to monitor the events collection for unprocessed outboxes via a recurring job. When an unprocessed outbox is found, the outbox repository returns the event data to Hangfire, stripped of the outbox information. Once Hangfire publishes the event, this repository will mark it as processed.

This solution guarantees at-least-once delivery, but as I explained in my previous post, all downstream systems are idempotent to cater for this.

Ashraf Mageed

EventSourcing, DDD and Microservices

Originally published at http://www.ashrafmageed.com on May 13, 2019.