Hello,
We have an (rare) issue when our service shuts down. It seems that the unitofwork for the command handling is always finishing but sometimes the event handler is not (or is not even started ?).
As a result the subscribing event processor (the default ) is sometimes missing events and the projection is not complete. These events will also never be picked up again on this projection.
Anyone else experienced this ?
Should we only use tracking processors for this reason ?
Gerlo.
After some debugging I found the spring shutdown hook calls EventHandlerRegistrar.stop which unregisters all subscribing event processors. This causes all commands which finish after the shutdown hook runs to persist their events without running those event processors.
Is this a configuration issue? Or maybe subscribing event processors should never be shut down?
Could be a issue with spring, shutting down services to early , but we checked it with latest spring versions , with the same results …
Nobody else having this issue ? This can be a serious issue when you have some query-store which has to be constistent (not eventually) with the events at al time.
Does someone at Axon has a thought on this ?
Spring allows you to control startup/shutdown order via the phase property on the Phased and SmartLifecycle interface.
https://docs.spring.io/spring/docs/5.1.2.RELEASE/spring-framework-reference/core.html#beans-factory-lifecycle-processor
It sounds like the subscribing event sources should have a higher phase than their handlers – so that sources are “started after and shutdown before” handlers.
Another option is to take control of starting your subscribing processors manually.
For example, in the app I’m working on there are some subscribing event sources that read from an external system.
These sources are configured not to autostart, and are started manually:
@EventListener(ApplicationStartedEvent::class)
fun startListeners(event: ApplicationStartedEvent) {
}
I haven’t tested whether this approach requires special handling to preserve events on shutdown yet.
However, setting the “phase” on components should work correctly?!
It would be great to get some more guidance on this issue… ty
Hi all,
I’ve had an offline discussion about this, but here’s a summary of it for the public record.
We have put an improvement for this on the backlog of Axon: https://github.com/AxonFramework/AxonFramework/issues/891. With this enhancement, Axon will change the order in which components are shut down, ensuring that existing processes are finished properly before shutting down these components.
In general, it is good practice to shutdown “edge” components first. Often, an application exposes an API. That component should be shutdown first, rejecting any new invocations, while allowing existing invocations to finish properly.
The Swiss railways have built a component that does this, by closing Tomcat connector in a Spring Boot application. It then gives the application a certain time to finish existing requests, before it continues with the actual Application Context shutdown.
Check it out here: https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown
Kind regards,
Allard