order of event handling for tracking processor?

Imported from Google Groups
1

I understand that by default SequentialPerAggregatePolicy is used.

“Event handlers may have specific expectations on the ordering of events. If this is the case, the processor must ensure these events are sent to these handlers in that specific order. Axon uses the SequencingPolicy for this. The SequencingPolicy is essentially a function, that returns a value for any given message. If the return value of the SequencingPolicy function is equal for two distinct event messages, it means that those messages must be processed sequentially. By default, Axon components will use the SequentialPerAggregatePolicy, which makes it so that events published by the same aggregate instance will be handled sequentially.”

So for the same aggregate, there are multiple events. How is the order decided among them? Each message comes with a token, from what I am seeing, it just takes a timestamp, does that mean the events for the same aggregate is sorted by time they arrived? In AbstractSequencedDomainEventEntry, the globalIndex is defined as below:

@Id
@GeneratedValue
private long globalIndex;

What is the use case of this globalIndex other than functioning as a primary key in the domain event entry table? I hope it is NOT used to determine the event handling order.

Thanks for the help in advance.

2

Hi Chun,

the global index defines the order of events in their global sequence. When events are inserted into the event store, they are assigned this sequence. When they are read, they are read using this sequence. In relational databases, there is a discrepancy between insert order (at which the sequence is defined) and commit order (at which the sequence becomes visible to other connections). To work around this issue, Axon uses GapAwareTokens. However, because Gaps can indicate temporarily missing events or rollbacks, the process won’t wait. These are events that were produced concurrently, and the order should therefore not have an influence on the outcome. Designed-for-purpose event stores don’t have this restriction.

When it comes to multi-threaded processing in Event Handler, events are made available to each thread in the order defined by the global sequence. However, which thread processes which event is defined by the SequencingPolicy. Still, evens with a lower GlobalSequence happened earlier and will be processed before events with a higher GlobalSequence.

Note that for EventSourcingHandlers, the global sequence is ignored. In that case, the aggregate-specific sequence number is used.

I don’t understand the background of your statement " I hope it is NOT used to determine the event handling order.". Why not?!

Cheers,

3

Allard,

Thanks so much for the quick and clear explanation. “Note that for EventSourcingHandlers, the global sequence is ignored. In that case, the aggregate-specific sequence number is used.” This confirms my understanding and is good. Global index is more like on a grand scheme, of course axon needs some order for different tracking processors. No issues here.

The concern I have is: by default, hibernate use oracle sequence to generate the @Id. We have two way replication between two databases. DBA does not like sequence. If I change to use UUID (controlled by our code) and change the globleIndex type from long to UUID, then as you were saying, there is no way to do reserve events for different tracking processors based on time without a whole lot change of axon code itself.

Thanks again for your time and help. Appreciated.

4

Allard,

One more question. Since globalIndex is the order used by tracking processor, and it is generated by oracle sequence, what if the sequence dose not properly increase? Say I have one active database where the sequence is all odd, so it increases like 1, 3, 5, 7, etc. Then I flip to the back up database (which is replicated), but sequence there is only even number starting with 2. So now the first globalIndex is 2, which is lower than the highest odd index already used. Will the GapAware token will be able to handle this? (How big the gap could be?). As you can see, the sequence will never repeat as they are even and odd, the concern is the relative value (new sequence is lower than the ones already used).

Or if you have other suggestion? Thanks.

5

Hi Chun,

The GapAwareTrackingToken can deal just fine with any format of gaps you will create due to (may I say so) terribly configured sequence generators (based on the example you are giving here).

It will however not detect that this new “old” sequence (as it start at 2) is an incorrect index.
Instead, it will think the gap was finally filled; even though this occurred due too a, and I say it again, “terrible” database configuration.

If you are highly concerned with this hypothetical scenario, I would suggest to put extra effort in the database configuration.
Or better, use Axon Server, as this will ensure you will have no gaps, ever.

Cheers,
Steven

6

Steven

Thanks for the reply. So the GapAwareTrackingToken thinks the gap was finally filled. But from execution perspective, any real impact? Say the processor is executing 1,3,5, etc. till suddenly it sees 2,4,6 etc. From processing event perspective, is there any real problem because all new events are now using even sequence.

I am still confused about how the processor is using the token. A specific processor is reserving events (segment) using the token. So it finishes 1,3,5 then starts processing2,4,6 etc. I need to read more docs.

Regarding the sequence design, looking for suggestions. We want to flip database instantly. Not sure if we can set the sequence starting value correctly during hot flip.

7

Hi Chun,

Whether it’s an issue from the event processing side, is more so a question you should target towards your Event Handling Components.
The Event Processor in Axon however does not particularly care about this as, as said, it will view these events with indices 2,4,6 to fill up the gaps that were left by your earlier sequence generator.

Note though that the framework has logic in place to disregard and/or clean up gaps in a given token.
You should look at properties like the maxGapOffset, lowestGlobalSequence and gapTimeout, which all three have some JavaDoc in both the JPA and JDBC Event Storage Engine.

Regarding the sequence design, looking for suggestions. We want to flip database instantly. Not sure if we can set the sequence starting value correctly during hot flip.

Now I begin to understand where you question is angled towards.

Would’ve been good idea to share the concern upfront, might have be a little more to the point.

With flipping instantly, do you mean on a live system?

So, a given (set of) Axon Application(s) is running, all connected to the same event store database, and during run time you are going to pull out the database under it’s feet and replace it by something else?

Cheers,
Steven

8

Steven,

I thought I have said the concern in the reply to Allard’s response, sorry that I may have confused you. Our system has standby database with two-way replication. Whether you call it pulling out under its feet or not, the intention is to flip the database in case of failure without human intervention. The sequence can’t be replicated two way (at least that is what the DBA told us) I am pushing for one-way replication. But they wanted to use this even/odd setting. Thus I am looking for how this is handled in other prod system.

9

Hi Chun,

The greater picture starts to make more sense, so thanks for the background.
Another question does pop up though.

What kind of failure would warrant a Event Store database flip to occur?

From a storage perspective, we typically suggest to maintain your token_entry table along side the Query Models they update.
The Event Store would then be given it’s own database, segregated from your models.

Regardless of your answers for the above, I would in this scenario not use the even/odd sequence number set up, which I base on the following.
Let’s state your system runs for X amount of time, which has incremented the generator towards Event with Global Index 5001.
Then, the “database-should-be-flipped”-failure occurs, which technically resets/adjusts the generator towards an index of 2.

Using the even/odd set up, you will by definition create gaps constantly.
This imposes constraints on the processing speed, but as gaps are intended to be temporarily to cover for insert-commit differences, you will also “loss events”.

Let’s go back to the situation that the database, and thus the sequence generator, is flipped.
Regular processing of your application proceeds, which will lead to the publication of an Event with global index 2.
This event will be brought to a Tracking Event Processor, who’s Tracking Token is set to index 5001.
The gaps of this token, which would included [2, 4, 6, 8,…], has however already been cleaned as the processor did not expect Event number 2 to come in, ever.

So, relatively long story which hopefully sheds some light on your scenario.
Simply put, the odd/even set up does not seem to practical.
Given the assumption that your Sequence Generator can not be replicated, I’d rather try to investigate what this Event Store database failure is that requires a database-flip.

Lastly, the issues you’re trying to solve here all resolvable by using a dedicated event store like Axon Server.
Given the pricing of it, I would have to guess it’s cheaper to go that route then to fine tune your own database(s) and likely write code to resolve the problems Axon Server already solves.

Cheers,
Steven

10

“This event will be brought to a Tracking Event Processor, who’s Tracking Token is set to index 5001.
The gaps of this token, which would included [2, 4, 6, 8,…], has however already been cleaned as the processor did not expect Event number 2 to come in, ever.”

Steven,

Why would you say this? Isn’t the gapawaretrackingtoken supposed to handle the gap and not losing the event?

11

Hi Chun,

Well Chun, I am saying this because of the following.

The GapAwareTrackingToken is provided in the framework to resolve the occurrence that several consecutive threads are publishing events to your store.
Database have a difference when it comes to “insert order” and “commit order” of entries, which can lead to discrepancies in the order of events in respect to their global index.

This is however by definition a short term issue.
What you are trying to achieve is a resolution where the gaps are permanently in stored, something which the default Event Storage Engines are not designed for.

You can adjust the gapTimeout (thus the timeout after which they are cleared out) and the gapCleaningThreshold (defining the size of the gaps) to adjust the behavior if you want.
However, if you would follow the route of storing all gaps indefinite, note that the stored Tracking Tokens will keep growing indefinitely as well.
This will have massive performance impact on your system in the long run.

Thus, sticking to the default Event Storage Engine approach, which will maintain your gaps for a selective period of time, is recommended from our side.

I know we have had a call some months ago now, but I feel that your team is trying to solve issue with Axon Server solves immediately for you.
From a cost benefit perspective in your side, it might be beneficial to look at it again and verify whether this is a quicker (and potentially cheaper) solution to your problem at hand.

Kind regards,

Steven van Beelen

Axon Framework Lead Developer

AxonIQ
Axon in Action Award 2019 - Nominate your project