Token and timestamp inconsistency in Axon event store

Axon Server
,
1

Hi,

we are using Spring Boot 2.4.0 and Axon Framework 4.4.5 with Axon Server 4.4.5 in one of our services. Recently we have experienced an interesting constellation in our event store.

Here’s the scenario: we have a TAggregate providing some CommandHandler and applying two events EventA and EventB in a sequence using the ApplyMore interface:

# inside CommandHandler of TAggregate
apply(EventA).andThen(() -> if ... apply(EventB))

Then, we have a TrackingEventProcessor set up as external event handler for EventA, which sends command CommandC asynchronously to be handled in MAggregate:

# inside CommandHandler of MAggregate
apply(EventC)

Now what we ended up with in the event store is the following:

| token | event  | timestamp                |
|-------|--------|--------------------------|
| 11577 | EventB | 2021-07-22T11:52:19.405Z |
| 11576 | EventA | 2021-07-22T11:52:19.404Z |
| 11575 | EventC | 2021-07-22T11:52:19.448Z |

So how can it be that EventC has a lower token than EventA, while the order with respect to the timestamps is correct? Is there some configuration I am missing that would treat the two events from the TrainingAggregate differently than the one from the external handler? Why does the EventC from the async command enter the event store before EventA, despite it should be guaranteed that EventA is fully processed and persisted in the event store, before the external event handler can see it?

What are we missing? Thanks for some thoughts on that!

Cheers,
Philipp

2

Hi @phil,

That’s quite interesting and triggered us to write a small demo to try ourselves.
So, basically it can only occur if the Event Processor is a subscribing one - if you are using a Streaming one (either tracking or pooled), it won’t happen.

I see you said it happened while using a TrackingEventProcessor so I would like to ask you a couple of questions:

  1. Was it a one of or can you reproduce it?
  2. Is it a local/dev environment?

Reason for the questions are that if you can reproduce it, we can easily track the problem. For the other, if it’s a local/dev env, it could be that someone was playing around with configs and it happened while using the subscribing mode.

Looking forward for your reply!

Kr,