How to handle many commands and events in a short time?

Imported from Google Groups
1

Hi,

First of all, we’re using MongoDB for the Axon store (domain events, tracking tokens, sagas) and the same mongo instance for our views/projections.

Our (migration) scenario is this:
We need to migrate employers and their employees from the legacy systems to our new Axon based system. We do this by “enrolling” (i.e. sending an enrollEmployerCommand) an employer. Upon this, we get (via SOAP services) some basic info on the employer, create the employer aggregate and update the view (via events). This is handled by a “employer” saga. Also, in this saga we get the list of employees for this employer (also a soap web service call). And for each employee in the response, we send an “enrollEmployeeCommand”.
This command creates the Employee Aggregate and starts a employee saga to fetch the details of the employee.

This results in a lot of events created in a very short time and we see MaxWaitQueueSize exceptions for MongoDB. We can allow for bigger wait queues, but I feel like this is just postponing the same issue.

`

EnrollEmployeeCommand resulted in an error: An event for aggregate [294c4597-632d-4ec5-a934-c31bf26a4f22] at sequence [0] could not be persisted, {}
com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded.

`

So I’m looking for some advise on how to handle these kinds of “migration” scenarios where we get a “storm” of commands in a short time, and we cannot affort to loose any of the events/commands. Any tips on how to balance the number of events created in a short time and tuning the connections to the event store would be greatly appreciated.

Thanks,

Danny

2

Hi Danny,

First off, sorry for keeping you waiting so long. We’ve been quite busy over here and sadly, sometimes something falls between the cracks…

Now, for you issue at hand.

A migration scenario like your sketching here typically does not have a one-off solution generally.
I would like to add that ‘a storm of commands’ should always be doable in your application, as there’s a chance this might occur outside of migration scenarios as well.

That said, I’d say tweaking your MongoDB to coop with the number of events being stored would be the ideal starting point for this, as it will future-proof your application as well.

If this is however not an option in your current system, I’d probably try to add some form of throttling on the enrolling process.

I assume from your description this is an action you can control yourself, thus it should allow you to not send all the actions in quick succession.

I haven’t used the MongoEventStorageEngine in a production environment myself, so I am sadly hard pressed to give you any tuning-tips in that space.

Maybe somebody else on the board can fill in some blanks here.

Hope this helps you out Danny!

Cheers,
Steven