Hello
does anybody have experience with eventstore which is “huge”? I mean billion or even trillion of events.
What kind of storage is sufficient for this? sql/nosql?
What could be the mechanism for loading the aggregates from this storage? I was thinking about storing snapshot and then moving old events to some archive. But in this case I don’t know how to perform replays.
What about primary keys?
We are currently using postgre/oracle db and generating primary keys using UUID4. The PK with UUID4 can have performance issue while it’s stored as string (binary representation could help here but axon with jpa is not ready for this maybe jdbc could help here ). Are you using other type of primary/unique keys?
during the Axon 3 Launch Event (where the upcoming Axon 3 release was announced), a developer of Casumo (casumo.com) gave a presentation about their applications. They have a pretty massive event store, and have overcome (many of) the issues that arise with big data sets.
There is a recording of the presentation. Unfortunately, we experienced some technical issues, so the sound quality isn’t optimal: https://www.youtube.com/watch?v=BHQEG–Pdwo
Basically, they append all their events to a table, which is partitioned. They have an archiver that writes events to a cheaper storage, such as filesystem, which is good at sequential reeds, but isn’t especially optimized for aggregate lookups. However, this data is still accessible in the production environment.
Every “now and then” they create a new partition for the application to write to, and drop the old ones. Before dropping it, they ensure that all aggregates mentioned in that partition have an updated snapshot, so that the data in that partition isn’t likely to be needed.
In some cases, it is necessary to recreate snapshots for an aggregate based on their events only. In that scenario, the archive can still be used to read those events.
Hope this gives you a bit of an indication.
Cheers,