Saga using PooledStreamingProcessor Token not moving ahead

Axon Framework
,
1

Hello
I’m running Axon Framework 4.10.1 on Kubernetes with multi module pods and running into a weird situation on one of our Sagas, with no clear errors logged.

The Saga has 31 segments. The Saga has been split into multiple segments for parallel processing since there was a backlog of ~ 110M events to be processed, based on the saga processor token entry and the max global_index in the domain_event_entry table.

The issue that I’m facing is that, a certain Kubernetes instance has multiple segments to be handled. The segment with the highest token entry is not moving ahead, until the other segments are caught up with the one that has the highest token.

This makes the parallel processing not useful, since the processor is just idle until the other segments are caught up.

saga_stuck_ss
saga_stuck_ss1149×130 25.8 KB

In the above image, token entry table has been filtered for one owner that handles 3 segments.
The only segment that’s moving ahead is the segment 6, The other two are just stuck.

Thank you.

2

This is my saga processing config

Function<String, ScheduledExecutorService> coordinatorExecutorBuilder =
        name -> Executors.newScheduledThreadPool(
            1,
            Thread.ofVirtual().name("[PSP] Coordinator - " + name, 0).factory()
        );
Function<String, ScheduledExecutorService> workerExecutorBuilder =
        name -> Executors.newScheduledThreadPool(
            100,
            Thread.ofVirtual().name("[PSP] Worker - " + name, 0).factory()
        );
    EventProcessingConfigurer.PooledStreamingProcessorConfiguration pspConfigSaga =
        (config, builder) -> builder
            .coordinatorExecutor(coordinatorExecutorBuilder)
            .workerExecutor(workerExecutorBuilder)
            .initialSegmentCount(2)
            .batchSize(100)
            .claimExtensionThreshold(10000);