Part of my application involves talking to an external service with a high per-interaction overhead that accepts work in batches. The work is typically triggered by client requests that themselves arrive in batches, either literally (a list of work items in a single message over the network) or in the form of multiple individual requests in rapid succession.
My intended solution is to publish a “queue a job” command to the command bus for each aggregate root that needs to talk to the external service. The command handler adds each command to a work queue, writes it to persistent storage so it’ll survive server restarts, then returns a “success” result. There is another piece of code that listens for additions to the work queue. As soon as it sees its first piece of work, it waits either until a certain amount of time has passed or the queue has reached a maximum batch size, pulls the commands off the queue, and fires off the interaction with the external service, publishing a “sent as part of a batch” event to Axon’s event bus for each queued command so the sagas associated with the aggregate roots can transition to “waiting for the external service” states. When the external service finishes the results are published as events on the event bus and the sagas pick those up and issue additional commands to trigger appropriate business logic.
Axon’s built-in support for queuing of messages seems to be tied to the distributed event bus and to require setting up a separate message broker such as RabbitMQ, which is something this application doesn’t otherwise call for and which seems like it’d add a bunch of unnecessary complexity.
Is there another approach I should be looking at?
One Axon-centric implementation I considered was to define an “interact with the external service” saga that’s not associated directly with any of my existing aggregate roots (it instead maintains a collection of pending work items, though each of those items does contain an aggregate root reference) and uses the built-in persistence and timer features of sagas to do its work. So basically I’d be using Axon’s saga management to implement a persistent timed job queue. This seems like it might work but feels like it violates the spirit of sagas in that it isn’t really representing a business process over a related set of aggregate roots. But I’m new enough to CQRS and Axon that my intuition about the spirit of sagas is not well-developed yet, and maybe this is actually a completely appropriate use of them.
Thanks in advance for any ideas or pointers to existing discussions of this topic!