Replaying Events on a Cluster


I’m trying to understand how “Replaying Events on a Cluster” can be done in a microservices architecture where events are persisted in different stores across different machines.

In order to repopulate the read model events should be replayed in the right order from the right store. Can this be done if in a distributed architecture we cannot rely on a single universal time ?



Hi Alessandro,

in general, the order of events from a single aggregate is the only fully guaranteed order. And of course events that were caused by another event, should generally be ordered as well. The order between all other events is not completely guaranteed.

In a replay from multiple source, I would recommend an approach where you read a message from each source, pick the one with the lowest timestamp (and then some other values as tie breakers) and process that one. Time on different machines isn’t guaranteed to be completely equal, but that’s generally not a problem, because the events have no absolute order either. Read the next from that source and compare again.
That said, my experience in general is that multiple components use the same “Event Store” to read messages from. In that case, your replaying component could just read from that store.



Hi Allard,

If understand correctly your reply, the following rules of thumb should be applied when implementing:

  1. Implement the read model so that it can be created from events broadcast by a single aggregate. Doing so will guarantee that it can rebuilt by replying the event in the original order.
  2. If the read model must rely on events from multiple aggregates, make sure that the order of events is not important (e.g. no temporal calculation like KPI, scheduled events, etc.)
    Would you agree and/or add anything to the above?



I would formulate the rule as:

if you depend on events from more than one consistency unit (either different aggregates or even when combining events from multiple instances of the same aggregate), make sure that you can cope with events coming in in another order than the events really occurred. Since the events are generated from different processes, there is absolutely no guarantee about ordering.