Got invalid event order (GenericEventMessage)

I got invalid event order which is published by EventBus inside of saga. here’s how it should works.

class GoogleCloudInitializationSaga {
  
  lateinit var eventBus: EventBus
  lateinit var commandGateway: CommandGateway
  
  lateinit var cloudServerId: String
  
  @SagaEventHandler(...)
  fun on(event: GoogleCloudServerDeviceSetupEvent) {
    deviceSetup()
    publishProcessingEvent(TaskState.DeviceMapping)
    // this will publish GoogleCloudServerLaunchingEvent further
    commandGateway.send<>(LaunchGoogleCloudServerCommand(...)) 
  }
  
  fun on(event: GoogleCloudServerLaunchingEvent) {
    launchServer()
    publishProcessingEvent(TaskState.Launching)
  }
  
  // this one just keep track current setup state of a cloud server
  private fun publishProcessingEvent(taskState: TaskState) {
    eventBus.publish(GenericEventMessage(
      GoogleCloudServerProcessingEvent(
        cloudServerId = cloudServerId,
        taskState = taskState
      )
    ))
  }
}

so as you can see it should publish processing event with taskState DeviceMapping first and then publish processing event with taskState Launching and so on, but here’s what I got from event store.

(simplify format from axon server)

{
  token: 19129,
  payloadData: {
    taskState: "Launching"
  }
},
{
  token: 19133,
  payloadData: {
    taskState: "DeviceMapping"
  }
}

So the event order is invalid, DeviceMapping should come before Lanuching (sometimes it works sometimes it doesn’t). I’m not sure if it related to this issue Not right order of events during read by findTrackedEvents · Issue #1750 · AxonFramework/AxonFramework · GitHub since I’m using axonframework version 4.4.8 Would upgrading newer version solve this issue.

The mentioned issue is about the MongoDb Event Store solution the framework uses.
With newer versions of Java, the date-time format adjusted slightly, causing a potential incorrect event ordering in MongoDb Event Stores.
I’m confident that’s not the predicament you’re facing in this case, @Lincoln_Burrows.

What is happening is that the EventBus stages the publication of the event in the UnitOfWork (UoW).
You can refer to the UnitOfWork as Axon’s "transaction around every message.

As you are invoking the EventBus within a Message Handler, there already is an active UoW.
The EventBus implementation validates whether there’s a UoW active and thus stages the publication of events in the prepare commit hook.

Now, on to the CommandGateway that dispatches the command onto the CommandBus.
As you’re doing a CommandGateway#send operation, you’re detaching the command handling from the Sage Event Handling thread.

In most cases, I’d assume this causes the expected event order.
However, in some cases, the command handling task will be faster than the Saga task.
The fact the Saga Event Handling thread, after execution, needs to perform serialization and storage to the entire saga might be the reason here.


Having said that, there’s a takeaway message here.
Whenever you use the CommandGateway#send operation, you detach the handling operation from the thread invoking that method.
As such, you enter the world of asynchronous communication, not giving you any 100% certainties on ordering.

Now, to make this response hopefully more helpful, let me ask you something.
Do you need those events to be in the expected invocation order, @Lincoln_Burrows?
If so, adjusting the method calls somewhat might help.

In all honesty, though, not forming a hard requirement on the event order will help you later on. It might make writing the event handlers a bit tougher depending on the task they need to perform, but it gives you greater flexibility in the end.

Thank you for answering my question, so according to your question “Do you need those events to be in the expected invocation order” I would say yes since we have some kind of progress bar or loading bar that keep track the current setup state of a cloud server and display to the user.

Thanks for clarifying that, Lincoln.

With that in mind, I would recommend pulling the event publication and command dispatching apart, if that’s feasible.
So for example, have the current Saga Event Handler publish the events, and that’s it.
Then you introduce another Saga Event Handler, handling the GoogleCloudServerProcessingEvent, which dispatches the command.

Doing so ensures you the former event is stored before initiating the LaunchGoogleCloudServerCommand task.


As a side note, I would consider having a more specific event than GoogleCloudServerProcessingEvent. From my perspective, it seems that the TaskState contained in the GoogleCloudServerProcessingEvent dictates the event we’re dealing with. So, why not have a specific event containing that state/status description? Granted, I’m not your domain expert, so I might be making a wrong assumption here.


Let me know whether this sounds feasible to you, Lincoln!
And if not, we’ll try to find a different route to ensure the desired event ordering.