MultiTenantEventScheduler: AxonServerException: CANCELLED

Did you change something recently in AxonCloud?

Calling eventScheduler.schedule(duration, event) we received CANCELLED since yesterday (12.09.2024). The day before it was working properly.

We have context in shared cluster:

org.axonframework.axonserver.connector.AxonServerException: CANCELLED
	at org.axonframework.axonserver.connector.ErrorCode.lambda$static$26(ErrorCode.java:164)
	at org.axonframework.axonserver.connector.ErrorCode.convert(ErrorCode.java:201)
	at org.axonframework.axonserver.connector.ErrorCode.convert(ErrorCode.java:232)
	at org.axonframework.axonserver.connector.ErrorCode.convert(ErrorCode.java:221)
	at org.axonframework.axonserver.connector.event.util.GrpcExceptionParser.parse(GrpcExceptionParser.java:57)
	at org.axonframework.axonserver.connector.event.axon.AxonServerEventScheduler.schedule(AxonServerEventScheduler.java:141)
	at org.axonframework.axonserver.connector.event.axon.AxonServerEventScheduler.schedule(AxonServerEventScheduler.java:156)
	at org.axonframework.extensions.multitenancy.components.scheduling.MultiTenantEventScheduler.schedule(MultiTenantEventScheduler.java:112)
	at [sanitized]Saga.scheduleDeadline([sanitized]Saga.kt:268)
	at [sanitized]Saga.on([sanitized]Saga.kt:60)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:153)
	at org.axonframework.modelling.saga.SagaMethodMessageHandlingMember.handle(SagaMethodMessageHandlingMember.java:80)
	at org.axonframework.messaging.annotation.WrappedMessageHandlingMember.handle(WrappedMessageHandlingMember.java:64)
	at org.axonframework.tracing.TracingHandlerEnhancerDefinition$1.lambda$handle$1(TracingHandlerEnhancerDefinition.java:84)
	at org.axonframework.tracing.Span.runCallable(Span.java:132)
	at org.axonframework.tracing.TracingHandlerEnhancerDefinition$1.handle(TracingHandlerEnhancerDefinition.java:84)
	at org.axonframework.messaging.annotation.WrappedMessageHandlingMember.handle(WrappedMessageHandlingMember.java:64)
	at io.axoniq.console.framework.messaging.AxoniqConsoleHandlerEnhancerDefinition$wrapHandler$1.handle(AxoniqConsoleHandlerEnhancerDefinition.kt:53)
	at org.axonframework.messaging.annotation.NoMoreInterceptors.handle(NoMoreInterceptors.java:46)
	at org.axonframework.modelling.saga.AnnotatedSaga.lambda$handle$8(AnnotatedSaga.java:126)
	at org.axonframework.messaging.Scope.executeWithResult(Scope.java:111)
	at org.axonframework.modelling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:126)
	at org.axonframework.modelling.saga.AnnotatedSaga.lambda$handle$7(AnnotatedSaga.java:118)
	at java.base/java.util.Optional.map(Optional.java:260)
	at org.axonframework.modelling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:118)
	at org.axonframework.modelling.saga.AbstractSagaManager.doInvokeSaga(AbstractSagaManager.java:182)
	at org.axonframework.modelling.saga.AbstractSagaManager.lambda$startNewSaga$3(AbstractSagaManager.java:121)
	at org.axonframework.tracing.Span.runCallable(Span.java:132)
	at org.axonframework.modelling.saga.AbstractSagaManager.startNewSaga(AbstractSagaManager.java:118)
	at org.axonframework.modelling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:105)
	at org.axonframework.eventhandling.MultiEventHandlerInvoker.handle(MultiEventHandlerInvoker.java:91)
	at org.axonframework.eventhandling.AbstractEventProcessor.processMessageInUnitOfWork(AbstractEventProcessor.java:195)
	at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$1(AbstractEventProcessor.java:173)
	at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57)
	at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:67)
	at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
	at io.axoniq.console.framework.eventprocessor.metrics.AxoniqConsoleProcessorInterceptor.handle(AxoniqConsoleProcessorInterceptor.kt:66)
	at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55)
	at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$2(AbstractEventProcessor.java:174)
	at org.axonframework.tracing.Span.runCallable(Span.java:132)
	at org.axonframework.eventhandling.AbstractEventProcessor.lambda$null$3(AbstractEventProcessor.java:170)
	at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:92)
	at org.axonframework.eventhandling.AbstractEventProcessor.lambda$processInUnitOfWork$4(AbstractEventProcessor.java:166)
	at org.axonframework.tracing.Span.runCallable(Span.java:132)
	at org.axonframework.eventhandling.AbstractEventProcessor.processInUnitOfWork(AbstractEventProcessor.java:165)
	at org.axonframework.eventhandling.pooled.WorkPackage.processEvents(WorkPackage.java:320)
	at org.axonframework.eventhandling.pooled.WorkPackage.lambda$scheduleWorker$5(WorkPackage.java:281)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:831)

Hi,
we rolled out a version upgrade to our cluster, that now runs the latest LTS 2023.2.9

is the issue still occurring?
can you please open a support ticket (you should have access to it from your menu, after logging in) writing down the context name and application name?

Thanks a lot.

The problem is still the same.
According to ticket creation. Do I really need to create 3rd account to create a ticket? I logged into our cloud console with Google and when I clicked support from a menu I was redirected to support service where I am asked to log in or create an account to create ticket and can’t see an option just log with google

An account on the support platform allows you to follow your ticket status. But it’s not necessary, indeed. But at least you need to write me down some hints about your context name and application name, otherwise I have no way to identify it and focus on logs. If you are comfortable, you can write them down here as well.

Created a ticket and referenced to this disussion