Axon Framework 4.2 vs 4.1.1: change or bug in EventProcessing error handling?

I want to keep my EventHandler retrying when an exception is thrown in the (tracking) EventHandler. The default behaviour is to continue processing the next event in the EventHandler. I do not want that, so I have created the following ErrorHandler:

`
@Component
public class RetryErrorHandler implements ListenerInvocationErrorHandler {

private final Logger log;

public RetryErrorHandler() {
this(LoggerFactory.getLogger(RetryErrorHandler.class));
}

public RetryErrorHandler(Logger log) {
this.log = log;
}

@Override
public void onError(Exception exception, EventMessage<?> eventMessage, EventMessageHandler eventMessageHandler) throws Exception {
log.warn("EventListener [{}] failed to handle event [{}] ({}). " +
“Starting retry mode”,
eventMessageHandler.getClass().getSimpleName(),
eventMessage.getIdentifier(),
eventMessage.getPayloadType().getName(),
exception);
throw exception;
}
}
`

With Axon 4.1.1, the RetryErrorHandler is being picked up by Spring Boot, and Axon. Axon gets an exception in the error handler, causing it NOT to update the token. As a result, the event is being redelivered to my EventHandler until the EventHandler no longer throws an exception. This is exactly the behaviour I want.

With Axon 4.2 (I changed axon version as ONLY change), the behaviour is different. The RetryErrorHandler is still picked up by Axon (Good) and executed when my EventHandler throws an Exception (Good) but now, the tracking token of the event processor IS updated. This result in the event no longer being offered for retry. NOT what I want.

What is the explanation about this changed behavior? Is this by design? Is this an AxonFramework bug? How can I make axon framework 4.2 eventhandlers to retry forever until they no longer throw an exception?

This is my Event handler:

`
@Component
@ProcessingGroup(“retry-event-listener”)
public class RetryEventListener {

public RetryEventListener(@Autowired SampleService sampleService) {
this.sampleService = sampleService;
}

private final SampleService sampleService;

@EventHandler
public void on(SampleRetryAggregateCreated event) throws Exception {
sampleService.eventListenerMethod();
}
}
`

And this is the test of my RetryErrorHandler:

`
public class RetryErrorHandlerTest extends AxonIntegrationTest {
@Autowired
private SampleService sampleService;

@Test
public void testEventProcessor() throws Exception {
when(sampleService.mock().eventListenerMethod())
.thenThrow(new Exception())
.thenThrow(new IllegalStateException())
.thenReturn(“Ok”);
commandGateway.sendAndWait(new CreateSampleRetryAggregate(UUID.randomUUID()));

verify(sampleService.mock(), timeout(10000).times(3)).eventListenerMethod();
}
}
`

The test succeeds in axon framework 4.1.1 and 4.1.2, but fails in 4.2 (because in 4.2, the eventListenerMethod() is invoked only once)

`
@Component
public class SampleService {

private SampleService mockDelegate;

public SampleService() {
mockDelegate = Mockito.mock(SampleService.class);
}

public String eventListenerMethod() throws Exception {
return mockDelegate.eventListenerMethod();
}

public SampleService mock() {
return mockDelegate;
}
}
`

Hi Stijn,

I had to dig a little, by diff-ing the entire codebase of the master branch against tag 4.1.2, but it did get me in the right direction.
For 4.2, we have done some performance improvements in the Tracking Event Processor (TEP) regarding work with the Tracking Token.

Initially, processing a batch of events required a TEP to first extend it’s claim on the Tracking Token when it start processing and update the token once it’s finished.

Thus, from a database perspective, you would always have two round trips to the token_entry table for every batch of events.

To improve performance in this area we’ve adjusted the TEP to update the Tracking Token before processing events entirely, effectively letting it perform a single round trip to the table.

I am assuming it’s this performance improvement which is causing the discrepancy between release 4.2 and 4.1.2 you’re seeing.
To switch this off, you’ll at the moment have to adjust the way a TEP is build, by using the TrackingEventProcessor.Builder#storingTokensAfterProcessing.

However, there’s one caveat to all of this which puts me off, which you can see in the referenced method:

Set this processor to store Tracking Tokens only at the end of processing. This has an impact on performance, as the processor will need to extend the claim at the start of the process, and then update the token at the end. This causes 2 round-trips to the Token Store per batch of events.

Enable this when a Token Store cannot participate in a transaction, or when at-most-once-delivery semantics are desired.

The default behavior is to store the last token of the Batch to the Token Store before processing of events begins.
A Token Claim extension is only sent when processing of the batch took longer than the {@link TrackingEventProcessorConfiguration#andEventAvailabilityTimeout(long, TimeUnit) tokenClaimUpdateInterval}.

I am mostly pointing to the highlighted section here.
As you state that, even though exception is thrown, the Token is stored in it’s updated state, correct?

That leads me to believe that the Transaction Manager might not be configured correctly.

Would you be able to shed some light on your Transaction Management part?

By the way, you can replace a ListenerInvocationErrorHandler for the PropagatingErrorHandler provided by Axon too.
That will give the exact same behaviour (a part from the logging) as your custom RetryErrorHandler.

Hope we can figure this out Stijn!

Cheers,
Steven

I have no particular code for configuring transactions. I leave it all to spring boot, axon and their starters. I do not have JPA or hibernate in the classpath. I do evertything with plain JDBC and springs JdbcTemplate.

Using debug, looking at axon’s source and logging, I will try figure out what is actually happening on the transaction level. Feel free to point to me where (which class) Axon auto configures it with the transaction manager, then I can verify if this is actually happening in my project.

Thank you Steven for pointing me to transactional issues.

I have added the following test in my RetryErrorHandlerTest:

`
@Autowired
private TransactionManager axonTx;

@Test
public void axonHasConfiguredTheSpringTransactionManager() {
assertThat(SpringTransactionManager.class).isAssignableFrom(axonTx.getClass());
}
`

This test fails in both axon framework 4.1.2 and 4.2:

java.lang.AssertionError: Expecting <org.axonframework.spring.messaging.unitofwork.SpringTransactionManager> to be assignable from: <[org.axonframework.common.transaction.NoTransactionManager]> but was not assignable from: <[org.axonframework.common.transaction.NoTransactionManager]>

So no transaction manager is being configured. However, I can inject the SpringPlatformTransaction manager, so in the end, that configuration is there.

Looking at where Axon is configuring its TransactionManager, in its TransactionManagerAutoConfiguration:

I copy the code over in my project with a little modification: I commented //@ConditionalOnBean(PlatformTransactionManager.class)

`
@Configuration
@ConditionalOnClass(PlatformTransactionManager.class)
@AutoConfigureAfter(name = “org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration”)
public class TransactionAutoConfiguration {

@Bean
@ConditionalOnMissingBean
//@ConditionalOnBean(PlatformTransactionManager.class)
public TransactionManager axonTransactionManager(PlatformTransactionManager transactionManager) {
return new SpringTransactionManager(transactionManager);
}
}
`

Then my all my tests (also the ones regarding to Listener exception behaviour) are working perfectly with Axon 4.2.

But when I include the @ConditionalOnBean(PlatformTransactionManager.class) (uncommented), Axon configures the NoTransactionManager instance and my tests fail. Probably because the condition is evaluated before Spring has instantiated the bean.

The @ConditionalOnBean is causing wrong behaviour in axon configuration in this case.

So I get this working if I do add axons TransactionManager myself in my configuration, but I believe that, it was axons intention, to do that automatically.

H i Stijn,

Thank you very much for this detailed description.
I think your suggestion of an ordering issue is correct, where my main focus is pulled towards the class level @AutoConfigureAfter annotation.

From the copied over snippet you see that it’ll only wait for the HibernateJpaAutoConfiguration class, prior to running the TransactionAutoConfiguration.

Would you be able to track from which auto configuration class your PlatformTransactionManager normally resides?

With that information, I feel we have a fix in place to be scheduled for 4.2.1.

The effort is very much appreciated Stijn!

Cheers,

Steven van Beelen

Axon Framework Lead Developer

AxonIQ
Axon in Action Award 2019 - Nominate your project

Hi Steven,

I found that my PlatformTransactionManager
is configured in org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration
from module org.springframework.boot:spring-boot-autoconfigure-2.1.4.RELEASE
That is because I am not using spring data/jpa or hibernate for my views, just spring’s JdbcTemplate.

@Bean
@ConditionalOnMissingBean(PlatformTransactionManager.class)
public DataSourceTransactionManager transactionManager(
      DataSourceProperties properties) {
   DataSourceTransactionManager transactionManager = new DataSourceTransactionManager(  //point of creation
         this.dataSource);
   if (this.transactionManagerCustomizers != null) {
      this.transactionManagerCustomizers.customize(transactionManager);
   }
   return transactionManager;
}

When run in debug, and I place a breakpoint on line (//point of creation), my code stops there at spring context startup. This proves that my PlatformTransactionManager is created at this point.

It’s really nice that you take this as an enhancement and are willing to bring it in a next release! Thanks!

Stijn

Hi Stijn,

Thanks for investigating which auto configuration class Axon’s TransactionAutoConfiguration should wait for!

FYI, I’ve just added it in this commit message, which makes it part of the upcoming 4.2.1 release.
Thus, stay tuned for it’s introduction! :slight_smile:

Cheers,
Steven