Archiving the event store

Hi, We have been using axon for our cqrs based microservice applications since more than a year now. We currently have one of the bounded context with more than half a million events in its event store and still growing. Since the event store seems to be too huge we are looking for approaches to archive events to improve performance and reduce event store size. We currently thought of the below approach.

Having to force create snapshots for all the aggregates and then archive the existing events outside the database in a compressed format etc. Since we have the snapshots for every aggregate it would load up from the snapshot initially after the archiving. The events created from then on will still be in the event store and we can repeat the cycle every 500k events etc. In case we have any aggregate structure changes or in case we face a scenario where in we need to replay all older archived events we can load up the events store with the archived events when ever needed and create updated snapshots. The aggregate structure changes would not occur very often and is very rare scenario.

It would be great if I can get suggestions regarding the downside if any for this approach, side effects or a better approach if available.

FYI: We are using MongoDB and also trackingtokens for query side.

Hi,

you can either choose to archive using a batching approach or continuously after creating a snapshot. Whichever approach you take, when the structure of your aggregates change, you would either need to replay all events (including the archived ones) or create upcasters for your snapshot events.

Have you considered using AxonServer for storing your events? It doesn’t suffer from performance degradation as much as other types of databases generally do.

Kind regards,

Hi,

I saw on the reference guide page that Axon Server stores events as files with separate configurable directories for events and snapshots. Is that possible to offload events to cheaper storage from events directory (given they are covered by snapshots) without having to tell Axon Server about it?

Hi Sam,

the only way that may currently work, is to move the file to another location and create a symlink from the old location to the new. We haven’t tested the approach ourselves, but given the OS usually nicely hides the fact that symlinks are used, it will most likely.

In the upcoming version of AxonServer Enterprise (4.4), we will have this feature built into the core of the storage engine. AxonServer wil then automatically manage files to only retain a certain volume of data on its primary disks.

Kind regards,

That's great, thanks!

Hello Allard,
I do use AxonServer for event storage for a research project. How do I drop the event store or reinitialize it? Because right now, the events keep coming back for every new scenario I’m working on

Hi Kindson,

Axon Server SE can be started in development mode (https://docs.axoniq.io/reference-guide/operations-guide/setting-up-axon-server/development-mode)

These features can be enabled by configuring the following property: axoniq.axonserver.devmode.enabled=true

Alternatively, you can delete the data folder from the axon server root folder.

Best,

Ivan