Processing group stuck and not processed

Axon Framework
1

Hi
I have a new projection with a new processing group in my Axon application (Axon v4.8.0, Axon Server v4.6.11). I defined 3 segments for the processing group and batchSize 10 for the event processor:

TrackingEventProcessorConfiguration
        .forParallelProcessing(3)
        .andBatchSize(10)

After deploying the new version of the app (2 instances in Kubernetes), the event processor started working and processed >11 million events (I can see this in the tokenEntry table), but not all events have been processed. Instead, the event processor “hang up” and got stuck. In Axon Server I see “Not all segments claimed” and number active segment = 0.

The tokenEntry table contains 2 rows for the processor for segments 1 and 2. There is no row for segment 0:

...ProjectionProcessor,2,,2023-08-02T22:50:30.138Z,org.axonframework.eventhandling.GlobalSequenceTrackingToken,"{""globalIndex"":13225850}"
...ProjectionProcessor,1,,2023-08-02T22:51:10.154Z,org.axonframework.eventhandling.GlobalSequenceTrackingToken,"{""globalIndex"":11769807}"

What is going on? Why is the event processor not picked up by any application. Restarting the apps did not change anything.

Any help is appreciated

Klaus

2

The tracking event processor should start three threads to handle all the events. That doesn’t seem to work in your case. Do you see something like Dispatching new tracking segment worker: {} in the logging?
The specific part that should create the new Thread is:

if (availableThreads.decrementAndGet() > 0) {
    logger.info("Dispatching new tracking segment worker: {}", trackingSegmentWorker);
    spawnWorkerThread(trackingSegmentWorker).start();
} else {
    workingInCurrentThread = trackingSegmentWorker;
    return;
}

Unless you set a thread factory, which doesn’t allow creating more than one thread, I don’t see how this could fail.

3

Hi Gerard,
thanks for your answer. Now I investigated the logs a bit more in detail: There are ""Dispatching new tracking segment " messages. And there are also message like

Fetch Segments for Processor 'xxx' failed: Could not open JPA EntityManager for transaction. Preparing for retry

Fetching Segments for Processor 'xxx' still failing: Could not open JPA EntityManager for transaction. Preparing for retry in

At that time we had issues with our Postgres database, so that no connection were available. This is fixed now and the event processor should continue working again. But this does not happen :frowning:

I also see Shutdown state set for Processor 'xxx'.in the logs. But how to restart after shutdown?

Klaus

4

There is still no token entry in the table, so it won’t pick it up. So to fix it, you need to either add the missing segment in the table, or remove al of them. Ideally such a situation should not occur, but that’s likely due to the PostgreSQL connection issues.