Axon FW 4.6.x introduces regression in Saga/Quartzscheduler

Axon Framework
,
1

Hi there

On axon 4.5.15 (with AS 4.5.12) everything works fine.
Spring Boot 2.7.5.
JDK 11.

Upgraded to axon fw 4.6.2.
Introduced axon-tracing-opentelemetry in combination with v. 1.18.0 of the opentelemetry agent (matches axon-tracing-opentelemetry).
We did not change AS.
We did not change Quartz tables (quartz version remained the same afaik).

At first sight, everything works fine, incl. tracing.
Until the moment our saga receives an event, and the injected eventhandler tries to schedule an event:

java.lang.IllegalArgumentException: Cannot dispatch new events as this scheduler is being shutdown
                at org.axonframework.common.Assert.isTrue(Assert.java:56)
                at org.axonframework.axonserver.connector.event.axon.AxonServerEventScheduler.schedule(AxonServerEventScheduler.java:135)
                at org.axonframework.axonserver.connector.event.axon.AxonServerEventScheduler.schedule(AxonServerEventScheduler.java:159)
                at be.acerta.ce.calendar.timespend.backend.adapter.connect.integrationmodel.usecase.assignconnectschedule.AssignConnectScheduleSaga.postponeWhenDataNotYetAvailable(AssignConnectScheduleSaga.java:252)
                at be.acerta.ce.calendar.timespend.backend.adapter.connect.integrationmodel.usecase.assignconnectschedule.AssignConnectScheduleSaga.on(AssignConnectScheduleSaga.java:215)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:153)
                at org.axonframework.modelling.saga.SagaMethodMessageHandlingMember.handle(SagaMethodMessageHandlingMember.java:80)
                at org.axonframework.messaging.annotation.WrappedMessageHandlingMember.handle(WrappedMessageHandlingMember.java:64)
                at org.axonframework.tracing.TracingHandlerEnhancerDefinition$1.lambda$handle$1(TracingHandlerEnhancerDefinition.java:84)
                at org.axonframework.tracing.Span.runCallable(Span.java:103)
                at org.axonframework.tracing.TracingHandlerEnhancerDefinition$1.handle(TracingHandlerEnhancerDefinition.java:84)
                at org.axonframework.modelling.saga.AnnotatedSaga.lambda$handle$8(AnnotatedSaga.java:121)
                at org.axonframework.messaging.Scope.executeWithResult(Scope.java:111)
                at org.axonframework.modelling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:121)
                at org.axonframework.modelling.saga.AnnotatedSaga.lambda$handle$7(AnnotatedSaga.java:113)
                at java.base/java.util.Optional.map(Optional.java:265)
                at org.axonframework.modelling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:113)
                at org.axonframework.modelling.saga.AbstractSagaManager.doInvokeSaga(AbstractSagaManager.java:181)
                at org.axonframework.modelling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:98)
                at org.axonframework.eventhandling.MultiEventHandlerInvoker.handle(MultiEventHandlerInvoker.java:91)
                at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$1(AbstractEventProcessor.java:173)
                at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57)
                at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:67)
                at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
                at org.axonframework.messaging.interceptors.LoggingInterceptor.handle(LoggingInterceptor.java:85)
                at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
                at org.axonframework.tracing.Span.runCallable(Span.java:103)
                at org.axonframework.eventhandling.TrackingEventProcessor.lambda$new$3(TrackingEventProcessor.java:185)
                at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
                at org.axonframework.eventhandling.TrackingEventProcessor.lambda$new$1(TrackingEventProcessor.java:179)
                at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
                at org.axonframework.eventhandling.AbstractEventProcessor.lambda$processInUnitOfWork$2(AbstractEventProcessor.java:184)
                at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:92)
                at org.axonframework.eventhandling.AbstractEventProcessor.processInUnitOfWork(AbstractEventProcessor.java:166)
                at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:490)
                at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:318)
                at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1145)
                at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.cleanUp(TrackingEventProcessor.java:1340)
                at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1317)
                at java.base/java.lang.Thread.run(Thread.java:840)

Downgraded to 4.5.15.
Removed OTel agent and axon-tracing-opentelemetry dependency.
Everything works again.

2

Hi Christian,

there was an issue with 4.6.1 with the lifecycle of certain components not being triggered correctly, but they have been addressed in 4.6.2.

You mention that you are using Quartz to schedule events, yet the stacktrace shows the AxonServerEventScheduler.

Could you perhaps share how you’ve configured the quartz event scheduler?

3

Hi Allard !

I was in the assumption that we were using the Quartz scheduler. But I only see now that this is not the case anymore – I didn’t notice that, shame on me.
The only thing we did is upgrade Axon fw to 4.6.2 and enable tracing/opentelemetry.
We did not change the existing quartz-config.

So we have:

In maven:

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-quartz</artifactId>
    </dependency>

In our spring config:

@Bean
public DeadlineManager deadlineManager(Scheduler scheduler, ScopeAwareProvider scopeAwareProvider, TransactionManager transactionManager, Serializer serializer) {
    return QuartzDeadlineManager.builder()
            .scheduler(scheduler)
            .scopeAwareProvider(scopeAwareProvider)
            .serializer(serializer)
            .transactionManager(transactionManager)
            .build();
}

In our application.yml:

# Axon Quartz
spring.quartz:
  job-store-type: jdbc
  jdbc:
    initialize-schema: never
  wait-for-jobs-to-complete-on-shutdown: true
  properties.org.quartz:
    threadPool:
      class: org.quartz.simpl.SimpleThreadPool
      threadCount: 10
      threadPriority: 5
      threadsInheritContextClassLoaderOfInitializingThread: true
    scheduler:
      rmi:
        export: false
        proxy: false
      instanceId: AUTO
      instanceName: MY_JOB_SCHEDULER
    jobStore:
      useProperties: false
      isClustered: false
      tablePrefix: ${spring.jpa.properties.hibernate.default_schema}.QRTZ_
      driverDelegateClass: org.quartz.impl.jdbcjobstore.MSSQLDelegate

I’m a bit confused, because I see that Axon fw 4.5.x also supports the AxonServerEventScheduler. However there – maybe also wrongly configured – we do not run into issues.
I assume that the axon autoconfig just made it work, while we where thinking that Quartz was taking care of scheduling etc in the background. While it maybe already was Quartz.

I will remove quartz completely and see if the problem persists.

4

So, this is what I did:

I did NOT remove quartz DeadlineManager. Deadline mgmt is something different than EventScheduler, something I first didn’t know. The EventScheduler is just the standard AxonServerEventScheduler, and the DeadlineManager uses Quartz.

To make everything working as expected, I upgraded AxonServer from v4.5.12 to v4.6.7. That’s it.