Hi,
I’ve been browsing the source code of Axon Framework’s Event Store persistence layer, to find out whether a common but little-known concurrency issue applies here that possibly results in missed events for consumers.
Namely, most implementations of Event Stores that I’ve encountered rely on a global serial number for events, and a “token” saved somewhere that tracks the consumer’s progress relative to that serial number.
This assumes that events become visible in the serial order, that is, the following invariant holds:
Given Domain Events a and b
If a.globalIndex < b.globalIndex
Then a was visible earlier or at the same time as b
This basically means that the Event Store prevents this sequence of occurrences:
- a.globalIndex is allocated a value (say, 1)
- b.globalIndex is allocated a greater value, as it should (say, 2); whether it has gaps relative to a’s index or not (as would happen with PostgreSQL’s sequence cache pre-allocating numbers) is unimportant
- b is committed and made visible to SELECTs
-
a is committed and made visible to SELECTs
For purposes of discussion, I call it the “sequential visibility principle”, and the situation above would violate it it.
This principle is not ordinarily true in SQL databases: it is possible for transactions to commit in any order, which is totally independent of order of sequence number allocation. This is because sequences are unaffected by transactions.
It is trivial to cause this by hand using 2 separate connections to a database (connections are marked as a and b, assume seqnum is a BIGSERIAL):
- a: BEGIN;
- b: BEGIN;
- a: INSERT INTO seqnums (seq) VALUES (DEFAULT);
- b: INSERT INTO seqnums (seq) VALUES (DEFAULT);
- b: COMMIT; – note: we run this on connection “b” first, violating the invariant as described
- a: COMMIT;
Causing a similar scenario, but in an Event Store, has an undesired consequence: when token-based consumers “tail” the Event Store, they may skip the earlier sequence number if it becomes “visible” to other transactions (in the MVCC sense) later.
Consider Domain Events with these sequence numbers present at some moment, t0:
[ 1, 2, 3 ]
Now, we append two events from totally separate entities. However, assume they COMMIT in the reverse order (which is totally fine by the database).
There exists a moment in time, t1, where the Event Store looks like this:
[ 1, 2, 3, 5 ]
Obviously, after some time, the transaction for sequence number 4 will end up committing and making the entries visible for SELECTs, or rolling back, in which case it is not a concern.
However, if at moment t1 a token-based consumer were to cycle through the Event Store, upon reaching number 5 it must think it has read the entire thing. Therefore, it will save its last-seen token (5), and will never go back to 4, at least under these assumptions.
My question is this: since Axon with the JPA-based Event Store relies on the globalIndex (BIGSERIAL) column for its tokens, what exactly prevents the scenario outlined above from happening? Do we employ locking (table locks or advisory locks) anywhere?