Issue with Retry Mechanism When Using Dead Letter Queue in Axon with Spring Boot

Axon Synapse
, ,
1

I am currently working on a Spring Boot application using Axon and have encountered an issue while implementing both retry on event processing failure and a Dead Letter Queue (DLQ).

In my configuration, I have registered two beans:

  • registerListenerInvocationErrorHandler – To handle errors and retry event processing.
  • registerDeadLetterQueueOnEventProcessingFailure – To store failed events in a DLQ after errors occur.

Issue:

When I include both configurations, the ListenerInvocationErrorHandler does not seem to work, meaning that my retry logic is not triggered before the event is stored in the DLQ.
However, if I remove the registerDeadLetterQueue bean, the ListenerInvocationErrorHandler works as expected.
Expected Behavior:
I would like to retry event processing up to 3 times before moving the event to the Dead Letter Queue if the error persists.

code:

@Bean
public ConfigurerModule eventProcessingForRegisterListenerInvocationErrorHandler(MongoTemplate mongoTemplate, RetryListenerInvocationErrorHandler retryListenerInvocationErrorHandler) {
    return configurer -> configurer.eventProcessing(
            processingConfigurer -> processingConfigurer
                    .registerListenerInvocationErrorHandler(
                            "customToolEventProcessor",
                            conf -> retryListenerInvocationErrorHandler
                    )
    );
}

@Bean
public ConfigurerModule registerDeadLetterQueueOnEventProcessingFailure(MongoTemplate mongoTemplate, RetryListenerInvocationErrorHandler retryListenerInvocationErrorHandler) {
    return configurer -> configurer.eventProcessing(
                    processingConfigurer -> processingConfigurer
                            // Configure Dead Letter Queue for unrecoverable errors
                            .registerDeadLetterQueue(
                                    "customToolEventProcessor",
                                    config -> MongoSequencedDeadLetterQueue.builder()
                                            .processingGroup("customToolEventProcessor")
                                            .maxSequences(256)
                                            .maxSequenceSize(256)
                                            .mongoTemplate(mongoTemplate)
                                            .transactionManager(config.getComponent(TransactionManager.class))
                                            .serializer(config.serializer())
                                            .build()
                            )
            );
}

@Component
public class RetryListenerInvocationErrorHandler implements ListenerInvocationErrorHandler {

    private static final int MAX_RETRIES = 3;

    @Override
    public void onError(@Nonnull Exception exception, 
                       @Nonnull EventMessage<?> event, 
                       @Nonnull EventMessageHandler eventHandler) throws Exception {
        UnitOfWork<?> unitOfWork = CurrentUnitOfWork.get();
        int retryCount = unitOfWork.getOrComputeResource("retryCount", k -> 0);

        if (retryCount < MAX_RETRIES) {
            unitOfWork.resources().put("retryCount", retryCount + 1);
            // Log the retry attempt
            System.out.println("Retrying event handling, attempt " + (retryCount + 1));
            throw exception; // Rethrow to trigger retry
        } else {
            // Log the failure
            System.out.println("Max retries reached for event: " + event.getPayloadType().getSimpleName());
            // Optionally, you can handle the event differently here, like sending it to a DLQ
            throw exception; // Rethrow to propagate the failure
        }
    }
}

Question:
Is there a recommended way to ensure that the retry mechanism is attempted before an event is stored in the DLQ? How can I configure Axon to first retry processing (e.g., 3 attempts) before sending the event to DLQ when all retries fail?

I would appreciate any insights or guidance on this issue.

2

Hi @Naib_Hossain, and welcome to our forum!

I can, sadly, be rather quick about your question.
When defining a dead-letter queue, you cannot customize the ListenerInvocationErrorHandler anymore. You should essentially regard the DLQ as a complete replacement of the ListenerInvocationErrorHandler layer for your Event Handling Components.

Although initially I hoped to have the DLQ be a ListenerInvocationErrorHandler, the API of the ListenerInvocationErrorHandler wasn’t broad enough to support dead lettering sufficiently. Because of this, we had to take the event handler wrapper class (called EventHandlerInvoker) instead.

If in doing so we kept support for the ListenerInvocationErrorHandler, we would give all kinds of possibilities to ignore the dead-lettering process entirely. Simply because the ListenerInvocationErrorHandler allows for a lot more than retrying (as you intend to do).

I did not like to take the decision to drop the ListenerInvocationErrorHandler support, but it was the safest way forward with the DLQ for most of Axon Framework’s users.

That said, if you do want to perform three retries, I think you will have to use the SequencedDeadLetterProcessor to immediately perform several retries in a row for the failing sequence identifier. Making it so specific might be quite some work, though. Hence, perhaps it is easier to simply use the SequencedDeadLetterProcessor#processAny perpetually in your application whenever an exception occurred with event handling.

To ensure retries occur trice, you would have to provide a custom EnqueuePolicy that adds a counter to the DeadLetter#diagnostics upon failure. Once this value hits 3, the custom EnqueuePolicy could decide to evict the letter.

I hope this helps, @Naib_Hossain! Be sure to post follow up questions if needed.