We have observed strange behaviour during import of historical legacy data into our axon (2.4.6) architecture. Basically what we are doing is importing documents, with all its modification and lifecycle state from a relational database into an Axon eventsourced model. Each document represents an instance of the DocumentAR, its historical data takes on average about 100 commands per AR to send, resulting in about 100-300 events.
Now initially this goes really fast, first couple of documents take a couple of hundred milliseconds. But after about 100 documents it takes already several seconds per imported document and even went up to 10-15s at which point we aborted the import process. We then did another run by importing the same document 200 times, and got the same result: first really fast then really slow.
Disabling the read side completely did not make a difference, so it’s really just a command dispatched to Axon into an eventsource aggregate over a distributed (jgroups) command bus. What was really strange is that it is even reproducible locally, with a single instance of the service using an H2 database!
Looking at some profiling data reveals not much, though i’m not sure what to look for. There was no GC pressure, database calls accounted for less than 5% of processing time, nothing else really stood out except for a huge amount of xstream objects which were all GC’ed when i forced garbage collection.
Any thoughts would be appreciated !