Sometime ago I wrote a implementation of the WS-Human Task specification using Axon (2.4.6), that basically uses Commands and Events on a Aggregate that represents a Task (eventually with sub-tasks) and it’s possible states and transitions (see picture)
So this of course rely heavily on the guarantee that commands and events are processed in the order they are invoked, for instance, if I send in succession the commands
Create
Activate
Suspend
I have to make sure the order of events raised correspond to the order of commands
Created
Ready
Suspended(Ready)
Failure to guarantee this will generate errors like ILLEGAL_STATE_FAULT or ILLEGAL_OPERATION_FAULT.
So to make a long story short this was working more or less fine for the last couple of years (a couple of ocasional errors here and there) until one month ago where those errors have grown to several per day. I do did some changes in the Axon configuration:
changed SimpleCommandBus to AsynchronousCommandBus
tweaked the commandBusTaskExecutor until finally set values of CorePoolSize=100, MaxPoolSize=200, QueueCapacity=100
changed ClusteringEventBus/DefaultClusterSelector/SimpleCluster to ClusteringEventBus/DefaultClusterSelector/AsynchronousCluster with a SequentialPolicy
tweaked the eventBusTaskExecutor until finally set values of CorePoolSize=100, MaxPoolSize=200, QueueCapacity=100
So my question is, will this configuration guarantee the order of processing of commands and of the events those commands generate?
with the AsynchronousCommandBus, it’s indeed possible for commands to be handled in a different order than which they were sent, especially with very large thread pools. In that case, multiple threads will take commands off the queue, and chase towards the lock. There is no saying which thread arrives at the lock first, because the OS will be doing a lot of thread switching in the meantime.
The best way to guarantee ordering of commands is by sending new commands as a result of the previous command’s result. Or you’d have to “tweak” the AsynchronousCommandBus and create a single task per aggregateIdentifier, where commands for the same aggregate would just be appended to an existing task.
The best way to guarantee ordering of commands is by sending new commands as a result of the previous command’s result.
Hmmm, in some cases I’m using sendAndWait when the pair of commands are always used in conjunction, like the Create and Activate, is that what you’re saying?
If you do “sendAndWait(createCommand)” and then “sendAndWait(updateCommand)”, you should get the expected ordering. There is no possibility of commands overtaking eachother.
What I really meant, was using send(createCommand).thenRun(() -> send(updateCommand)). This is a non-blocking approach that achieves a similar goal.
I did replaced all my send() by sendAndWait() in our main Saga, and the result was a disaster…
Very quickly the Saga part of the application froze without any external signal, no error, o time outs, nothing. I finally had to dump all the threads and saw the blocked threads on IdentifierBasedLock$DisposableLock.lock()
it’s not unlikely that using sendAndWait in a Saga causes a deadlock somewhere, depending on what your Saga does, exactly. That’s basically why I suggested:
send(createCommand).thenRun(() -> send(updateCommand))
That should work well, even with an asynchronous command bus.
I don’t think the trade-off is between performance and load. It’s more performance+load vs programming model. Adopting asynchrony all the way would be the way to go here.
