Hi
I have a problem with an event transformation that I started on an (staging) AxonServer cluster (v2025.2.7. The transformation applies upcasters to the event store and it worked on smaller event stores successfully.
Using the listTransformations() API I can see that the event transformation “f7a08b67-609f-44b4-8da8-bb12a20f3a89” has been started on 2026-06-03T16:45 and is still in APPLYING state since days. Unfortunately, I do not see any progress.
In two of three axon server nodes I see logs like this since days:
{
"timestamp": "2026-06-08T11:38:06.932+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applying transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:06.933+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Starting to apply transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 to context default",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:06.936+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 applied successfully to local store.",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:06.936+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applied transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:16.939+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applying transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:16.940+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Starting to apply transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 to context default",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:16.945+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 applied successfully to local store.",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:16.945+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applied transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:26.948+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applying transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:26.949+02",
"level": "INFO",
"thread": "boundedElastic-7",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Starting to apply transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 to context default",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:26.952+02",
"level": "INFO",
"thread": "boundedElastic-5",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.DefaultTransformationApplyExecutor",
"message": "Transformation f7a08b67-609f-44b4-8da8-bb12a20f3a89 applied successfully to local store.",
"context": "default"
}
{
"timestamp": "2026-06-08T11:38:26.953+02",
"level": "INFO",
"thread": "boundedElastic-5",
"logger": "io.axoniq.axonserver.eventstore.transformation.apply.TransformationApplyAction",
"message": "Applied transformation: f7a08b67-609f-44b4-8da8-bb12a20f3a89",
"context": "default"
}
So it looks like the nodes are still applying the transformation, but I cannot see any progress.
One node (axonserver-2) seems to be stuck and does not recover. Also, this node is not up-to-date with replication, Axoniq platform shows it as “Node is behind, 4413524 events have been committed but not applied”.
Here are logs that show up every minute:
{
"timestamp": "2026-06-08T11:58:45.256+02",
"level": "INFO",
"thread": "default-raftNode-0",
"logger": "io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalTransformers",
"message": "Invoking transformerFor default",
"context": "default"
}
{
"timestamp": "2026-06-08T11:58:45.257+02",
"level": "INFO",
"thread": "default-raftNode-0",
"logger": "AUDIT.io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalEventStoreTransformationService",
"message": "<anonymous>@default: Request to replace event 224216",
"context": "default"
}
{
"timestamp": "2026-06-08T11:58:45.257+02",
"level": "INFO",
"thread": "boundedElastic-8",
"logger": "io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore",
"message": "Finding transformation with id 73982957-04a6-4490-843c-e1723eaa6984.",
"context": "default"
}
{
"timestamp": "2026-06-08T11:58:45.257+02",
"level": "INFO",
"thread": "boundedElastic-8",
"logger": "io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore",
"message": "Found transformation with id 73982957-04a6-4490-843c-e1723eaa6984.",
"context": "default"
}
{
"timestamp": "2026-06-08T11:58:45.258+02",
"level": "WARN",
"thread": "boundedElastic-8",
"logger": "io.axoniq.axonserver.eventstore.transformation.requestprocessor.TransactionalContextTransformer",
"message": "Rolling back the event transformation transaction for context default. Reason: ",
"context": "default",
"exception": "io.axoniq.axonserver.filestorage.impl.FileStoreException: Cannot reset to value higher than the last index\n\tat io.axoniq.axonserver.filestorage.impl.BaseAppendOnlyFileStore.lambda$reset$2(BaseAppendOnlyFileStore.java:55) <<SNIP>>"
}
{
"timestamp": "2026-06-08T11:58:45.259+02",
"level": "WARN",
"thread": "boundedElastic-8",
"logger": "io.axoniq.axonserver.eventstore.transformation.requestprocessor.LoggingEventTransformationService",
"message": "There was an error while trying to accept replace action with event token 224216 for transformation 73982957-04a6-4490-843c-e1723eaa6984 and context default. The received sequence of the transformation action: 40698.",
"context": "default",
"exception": "io.axoniq.axonserver.filestorage.impl.FileStoreException: Cannot reset to value higher than the last index\n\tat io.axoniq.axonserver.filestorage.impl.BaseAppendOnlyFileStore.lambda$reset$2(BaseAppendOnlyFileStore.java:55) <<SNIP>>"
}
{
"timestamp": "2026-06-08T11:58:45.259+02",
"level": "WARN",
"thread": "boundedElastic-8",
"logger": "io.axoniq.axonserver.eventstore.transformation.cluster.logentryconsumer.ReplaceEventLogEntryConsumer",
"message": "Unsuccessfully accepted replace event [token 224216] action for transformation 73982957-04a6-4490-843c-e1723eaa6984 in context default.",
"context": "default",
"exception": "io.axoniq.axonserver.filestorage.impl.FileStoreException: Cannot reset to value higher than the last index\n\tat io.axoniq.axonserver.filestorage.impl.BaseAppendOnlyFileStore.lambda$reset$2(BaseAppendOnlyFileStore.java:55) <<SNIP>>"
}
The exception shortened above is the following:
io.axoniq.axonserver.filestorage.impl.FileStoreException: Cannot reset to value higher than the last index
at io.axoniq.axonserver.filestorage.impl.BaseAppendOnlyFileStore.lambda$reset$2(BaseAppendOnlyFileStore.java:55)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
at reactor.core.publisher.Mono.block(Mono.java:1779)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore.storeStagedActions(LocalContextTransformationStore.java:78)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore.save(LocalContextTransformationStore.java:63)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.TransactionalContextTransformer.lambda$perform$13(TransactionalContextTransformer.java:166)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.TransactionalContextTransformer.lambda$executeInTransaction$0(TransactionalContextTransformer.java:56)
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:61)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:237)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:241)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onNext(MonoSubscribeOn.java:146)
at reactor.core.publisher.MonoSupplier$MonoSupplierSubscription.request(MonoSupplier.java:145)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.trySchedule(MonoSubscribeOn.java:189)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onSubscribe(MonoSubscribeOn.java:134)
at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:48)
at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
*__checkpoint ⇢ Transformation updated after REPLACE_EVENT
Original Stack Trace:
at io.axoniq.axonserver.filestorage.impl.BaseAppendOnlyFileStore.lambda$reset$2(BaseAppendOnlyFileStore.java:55)
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:61)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:118)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2570)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.request(FluxFilterFuseable.java:191)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:135)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onSubscribe(FluxFilterFuseable.java:87)
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1864)
at reactor.core.publisher.MonoCacheInvalidateIf.subscribeOrReturn(MonoCacheInvalidateIf.java:170)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)
at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:335)
at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:294)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:373)
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
at reactor.core.publisher.Flux.subscribe(Flux.java:8891)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:196)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1864)
at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:151)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.Mono.block(Mono.java:1778)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore.storeStagedActions(LocalContextTransformationStore.java:78)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.LocalContextTransformationStore.save(LocalContextTransformationStore.java:63)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.TransactionalContextTransformer.lambda$perform$13(TransactionalContextTransformer.java:166)
at io.axoniq.axonserver.eventstore.transformation.requestprocessor.TransactionalContextTransformer.lambda$executeInTransaction$0(TransactionalContextTransformer.java:56)
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:61)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:294)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:237)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:241)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onNext(MonoSubscribeOn.java:146)
at reactor.core.publisher.MonoSupplier$MonoSupplierSubscription.request(MonoSupplier.java:145)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.trySchedule(MonoSubscribeOn.java:189)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onSubscribe(MonoSubscribeOn.java:134)
at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:48)
at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Any help is appreciated.
Klaus