We encountered a problem that will cause the read store to be inconsistent with the write store because of out of order events. Following described the problem and some options we thought to address it. But non of them seems good enough because they’ll add quite a lot of conflict resolution logic to the application code. Looking for better solutions.
We have 2 domains. Each domain is an axon application.
Domain1 has an event sourcing aggregate root that’ll change its status based on user requests. Those status change events are stored in eventstore and published to eventbus (JMS based).
Domain2 listens to the JMS event bus for those status change events by using EventListener. the EventListener will save the status to its read JPA based store (to be used by other logic).
The problem happens when, in Domain1, the status changes from A->B and then from B->C in very short time, the 2 events for those 2 changes go to the JMS. But when Axon+spring load the messages from the JMS and trigger the event listener, we found the events could be executed out of order. (The status change event only contains the changed value.) if B->C runs before A->B, the read side will end with status = B, which is inconsistent with write side (status = C).
It looks the event could be out of order because
- domain2 has a multiple machine cluster that listens on the same ActiveMQ queue and they runs in paralell
- JmsListenerContainerFactory could be configured to run multiple thread.
BTW, we use SimpleCluster, which should not causes concurrent problem itself.
- High water mark sequence number
Add logic in event listener. when process an event, comparing the event’s sequence number(or timestamp) with the last sequence number in the read store, if event’s sequence number is bigger, update the status read store with the sequence number; otherwise discard the event.
- Replay events to build latest read side
On receive a event, load all the events for the corresponding aggregate from the event store and replay it. then store the latest status in read store
- Include From/To status in event + retry
Add both the old status and new status. In the event listener, comparing the old status with the status in the store. if match, updated; if not, throw exception to cause the event to be retried later. And hope the earlier event could be processed during the wait time.
The problem could get more complicated if the read store is a denormalized view of events from multiple domains.
Looking forward for your thoughts.