We’re using Axon Server community edition and Postgres for our projections and JPATokenStore on an AWS EKS cluster - kubernetes and helm:
When we upgrade our helm charts causing AxonServer and our Axonified Spring Boot application (pods) to restart, we end up in a state where event handlers are no longer firing. In remote debug we can see that the commands and EventSourcingHandlers are being invoked properly, but none of our EventHandlers are ever invoked.
We seem to have some success by manually dropping the TOKEN_ENTRY table in the Postgres database. We are speculating that by restarting the AxonServer but leaving the TokenStore untouched, we are probably breaking the EventProcessor. But then again, we’re not deleting the PV between AxonServer restarts, so it’s unclear.
What should we expect when restarting our service and AxonServer?
Should we be dropping TOKEN_ENTRY tables whenever the AxonServer restarts?
Should we only do that when the AxonServer volume (disk space) is cleaned?
What are the best practices here?
Do you keep the storage for Axon Server? Seems a lot like a recent stack overflow issue. If you don’t keep the events, you probably also need/want to clean the database.
Thanks Gerard, I put the question on SO after a couple of days of no answer here - and hadn’t seen your answer.
To clarify though: What you’re saying is that the lifecycle of the database tables used for token tracking must match that of the event store. That is: if wipe one we must wipe the other, and vice versa, if we maintain one, we must maintain the other.
If so, that makes sense. We thought we were seeing different behavior but we’re doublechecking.
The related question then is: does Axon have APIs that help us in any way with this? E.g. Is there an API to reset the event store to zero programatically. And if so, is there a way to rest the token store accordingly?
(It wouldn’t have to truncate or drop tables, merely update or insert to set the token tracking back to zero)
If you enable developer mode, using env variable AXONIQ_AXONSERVER_DEVMODE_ENABLED=true, there is a button in the UI, there should be an API endpoint that does the same.
edit: the endpoint is a delete at http://localhost:8024/v1/devmode/purge-events (when running local).
As I also explained in the Stackoverflow issue, you probably don’t only want to clean the tracking tokens, but also any data based on the deleted events in the database. The stored token is basically just a number. So if it’s set at 100, it will only start reading again after there have been 100 events created.