Long running process: How to find out when projections are done?

Axon Framework
#1

Hello Axon nerds
My scenario looks simplified as follows:
I need to import a large number of objects, let’s say customer and order records. The import is a long running process consisting of multiple phases, e.g. first the customers are imported and afterwards the orders. After the import of the orders is finished I need to run some analysis on the imported orders.
My approach so far is that I trigger the import by a StartImport command and do the coordination of the long running import using a saga.

The whole import process works fine on the command side, everything runs smoothly in the background and the saga coordinates the different import phases by using different commands and events. On the command side, my import has a defined state and I can also signal the end of the overall process.

The problem is the last step, the analysis of the orders: This step needs to query the imported orders and must access the read model.

But how do I know when the eventual consistent projections of all imported orders are finished? I tried publishing an event after the last import order command to signal the end of import, but this event seem to be handled out of order and may “overtake” the projection events, at least it was not reliable. I am using TrackingEventProcessors and have a distributed setup.
It’s no problem that the projection of the orders take some time to get consistent, but I would like to have a reliable event that my saga can use, to trigger the analysis phase.

Somewhere else I am using a subscription query to get notified when the projection of a single object has been finished, but this approach simply does not feel right for this import process that deals with thousands of imported objects.

Any hints or ideas how to solve this kind of problem with Axon?

Klaus

Deleting lots of aggregates
#2

Hi Klaus,

I think you need the EventTrackerStatusChangeListener. You can find how to use it in our ref-guide. If you have any questions please let us know.

Best,

Yvonne

#3

Hi Yvonne
I checked the EventTrackerStatusChangeListener, but I don’t think that this gives me a reliable way to find out if the events I am interested in have been processed. The listener just tells me that the event processor has caught up. But the event processor might also be busy because of other parallel imports. So the listener just tells me “I am busy” or “I have caught up”.

Isn’t there some other trick to solve this using events and/or sagas?

One approach could be to add some special flag in the “OrderImported” event to signal the last order imported in the process and then fire a special “import finished” event, but this does not feel right as it mixes up process details with domain data in my opinion.

Klaus

#4

So if I understand you correctly you want to know if a particular batch of events have been processed?

#5

Yes, that’s correct.

The import processes batches of objects in multiple phases. I think such a requirement is not uncommon. But maybe I just did not get the right idea how to model this kind of batch processing with CQRS:-)

The import imports batches of some thousands of objects which may take some time. At the end I want to run this analysis task I described over the imported objects. And therefore I need to wait until all projections are eventually stored in the database.

#6

Yes, I agree it’s a common use case. What you did with the EndImportCommand/ImportEndedEvent is the right thing to do, you can use the getCurrentPosition from the event Processor to wait for it to be caught up with the position of the ImportEndedEvent.
To make this a little bit easier we are going to return the position of the last appended event:

I hope this helps

Yvonne

#7

Hi Yvonne
thanks for your answer.

Then I will try to use follow the approach using a dedicated “end of import phase”-command.

Klaus

#8

I would like to add something to this. We in general recommended to separate the concerns of command, event and query handling. A steppingstone towards this, is to embrace the eventually consistent nature of an Axon application. This suggest you do not actively query for data to be update, but are kept up to date.

This can be achieved by subscribing to query models instead of querying with a certain expectation.

This is what the framework provides the subscription queries for, actually.