First of all, many thanks for the great framework, I am new to Axon Framework and I feel I am already falling in love with it. In short, here is my setting and problem:
Setting
Axon Server SE.
Spring Cloud App as gateway, distributing incoming REST requests to mircroservices (Spring Boot Apps) via Axon Framework.
One of the microservices (so far the only one implemented) uses MongoDB for storing/retrieving documents. I have set up a MongoDB based token store and use the same MongoDB as for documents (as suggested in Axon Guide).
The âDocumentâ aggregate has proper 'CommandHandlerâs and 'EventSourcingHandlerâs for creating and updating documents.
A âDocumentProjectionâ service â a Spring Boot Service with ProcessingGroup(âdocumentsâ) â has several 'EventHandlerâs which are applied events by the Document aggregateâs command handlers, and care about maintaining the projection database, MongoDB it is, and retrieving documents via fetch events, accordingly.
The described setting above is nothing special, it follows the Axon examples such as giftcard, hotel reservation, etc. Now, here is my problem.
Problem
During the development I have refined the âDocumentâ aggregate and restructured things such as event classes. And somehow I came several times into the situation, that my event handlers in DocumentProjection service did not get invoked anymore. I have deeply dived into axon sources and could not find the reason. But I could make an observation: every time I ran into this problem, I deleted the trackingtokens collection in MongoDB and restarted the service, and suddenly all events began to be delivered as expected, also those, which I had missed when I created or updated documents which did not arrive before.
My Question
Is there something special I have to consider to avoid such a situation? It seems that the trackingtokens get out of sync with the axon server, and I could not understand, how I managed to break the trackingtokens. Could that have to do with playing with event classes?
Let me focus on one specific sentence in your question, as that stroke me as interesting:
This sentence to me seems like you were expecting Axon Framework to automatically rehandle events after you have adjusted their format.
Is that a correct assumption from my end?
If so, I can be short about it: Axon Framework does not automatically replay events once you have changed the structure.
If you do require a conscious replay of events, you should use Axonâs Replay API.
This would mean youâd first retrieve the correct StreamingEventProcessor (in your case, the processor with the name documents, as follows from the @ProcessingGroup("{...}") annotation) and invoke the StreamingEventProcessor#resetTokens operation.
This will let your processor start from the beginning of the event stream to replay your events.
Granted, the above only makes sense if my earlier assumption is correct.
So, feel free to add another reply stating whether this is the case.
This sentence to me seems like you were expecting Axon Framework to automatically rehandle events after you have adjusted their format.
Is that a correct assumption from my end?
Nope, what I meant was that new events issued as side effect in new incoming commands (using the AggregateLifecycleâs apply method) did not arrive in the corresponding event handlers. I was not interested in past event, but only in new ones.
I cleared also the event store (because I started the axon server in debug mode), and saw no change.
Point your application to a new event store?
No, the event store config was not changed.
Introduce upcasters for your events?
Nope. No event casters were used at all.
One of the things I have changed was changing the event class itself. I donât know if that was the reason for the problem.
I think, I have definitely messed up something and that explains the observation. I wished to get some logs from axon framework to know, what exactly I have messed. I have set the log level of axon framework to trace and still see no anomalies. Are there any other ways to increase the log details?
As far I am on the early development phase, it is not a big deal to erase the trackingtokens collection every time I see missing event arrivals. However, if I run into a similar problem later when the system is on production environment, then erasing the trackingtokens collection would be something I would like to avoid.
When you clear out the Event Store but do not change the stored tokens, your tokens point to a position your clean Event Store hasnât reached yet.
See it as such:
Your current event store contains 100 events.
If your Event Processors are on track, theyâve handled all 100 events.
This makes their tracking token point to position 100.
Now you stop your application, remove all events from the store and start it up again.
When new events are stored, theyâll start at position 0, incrementing from there.
However, for the Event Processors, it seems like theyâre already on track.
They are actually 100 events in front of the work!
Hence, they wonât show you the ânewâ events simply because the tokens dictate the processor has already processed those events.
This means that whenever youâre in a development phase, and youâre clearing out your Event Store, itâs recommended to clear out your query models and tokens too.
