We’ve encountered a challenge here at ihomer that we’re struggling to resolve ourselves, and unfortunately, Google hasn’t been able to provide any alternative solutions.
Our application is built on the Axon framework 4.7.6, and we’re operating without an event store, instead using a simple event bus. The application has multiple listeners to an event, each committing queries to a database using Spring Boot’s @Transactional annotations (with Hibernate serving as the JPA implementation).
The issue arises when one transaction in one of the listeners fails with an unchecked exception. This marks the transaction as dirty, and since the other handlers seem to be using the same transaction (which is now rollback-only), they also fail.
At this point, enabling the event store (and thus tracking event processors, right?) isn’t a feasible solution for us. However, we do want to commit the other queries from the other handlers that didn’t encounter an error. Essentially, we’re looking to have each listener pass or fail independently using transactions.
We’ve attempted to add (propagation = Propagation.REQUIRES_NEW) to each @Transactional above each handler. This seems to work, but it’s not an ideal solution.
We’re open to any suggestions or advice on what to try next.
Thank you in advance.
Also, if I’ve made any incorrect assumptions about the Axon framework, please feel free to correct me. I’m still relatively new to this framework.
When you say you are not using an event store, you mean you are not using Axon Server, right? I assume you do use the JpaEventStorageEngine?
Another question that pops up is if you only use subscribing event processors currently.
Towards a solution, you could configure an error handler, and catch it earlier, so the transaction won’t be rolled back. I’m not sure that would be much better than the current workaround you are using.
When you say you are not using an event store, you mean you are not using Axon Server, right? I assume you do use the JpaEventStorageEngine ?
No, unfortunately we don’t use either.
Another question that pops up is if you only use subscribing event processors currently.
Yes, we only use subscribing event processors. That’s our only option when not using the event store, right? (since there are no objects created implementing StreamableMessageSource)
Towards a solution, you could configure an error handler, and catch it earlier, so the transaction won’t be rolled back. I’m not sure that would be much better than the current workaround you are using.
Do you hint towards using a try-catch block or implementing a custom axon ErrorHandler?
I’m not sure where it would be best to catch the error. As it depends on which stage of processing the error occurs. If it’s related to business logic, a custom error handler will work. If it’s on the database level, as part of the commit of the whole transaction, this likely will not work. You probably also only want the partial commit only for some errors?
I’m not sure where it would be best to catch the error. As it depends on which stage of processing the error occurs. If it’s related to business logic, a custom error handler will work. If it’s on the database level, as part of the commit of the whole transaction, this likely will not work.
I’ve tried implementing a custom error handler, and whilst it made the other listeners go through with processing the event, when the transaction is dirty due to a previous error, the whole transaction isn’t committed. So unfortunately this does not work.
You probably also only want the partial commit only for some errors?
Ideally, yes, but if it works for any exception that will also fix our problem for now.
We know our axon configuration isn’t ideal, so thank you for taking the time to help.
I could think of one that’s also not ideal, which is changing the unit of work. I’ll ask if someone has a better idea. Did you notice any of the downsides linked in your first message?
Did you notice any of the downsides linked in your first message?
In addition to the potential challenges outlined in the link in my initial post, we’ll have to ensure that each time a transactional event handler is created, (propagation = Propagation.REQUIRES_NEW) is added when necessary. However, it’s inevitable that this may be overlooked at some point.
Chiming in here as part of Gerard requesting some input.
I am pretty confident you could resolve your scenario by setting a custom TransactionManager with Axon Framework, @joey-ihomer.
Axon’s TransactionManager interface is used by the UnitOfWork to:]
Start a Transaction, which can be committed and rolled back.
Add the Transaction#commit operation to the onCommit phase of the UnitOfWork.
Add the Transaction#rollback operation to the onRollback phase of the UnitOfWork.
The Axon Framework project at this stage provides two TransactionManager implementations:
NoTransactionManager - the default, as this ensure everything works out of the box.
SpringTransactionManager - the default whenever Spring is involved and a PlatformTransactionManager bean is present in the ApplicationContext.
As you’re using the @Transactional annotation, I am pretty confident we’re dealing with a Spring application. If that’s the case, you can override the SpringTransactionManager to, on the startTransaction operation, always require a new transaction.
If you then make your custom SpringTransactionManager bean part of the Application Context, Axon Framework will automatically pick it up and configure it wherever Axon uses a TransactionManager.
Do note that doing this means that you will create a new transaction for any transactional component in Axon Framework.
Whether that’s desirable isn’t clear for me, as I am not familiar with the outline of your project.
If it is not, you should consciously set this custom SpringTransactionManager for Event Processing only, which you can do through the EventProcessingConfigurer#registerTransactionManager(String, Function<Configuration, TransactionManager>) method for specific event processors, or through EventProcessingConfigurer#registerDefaultTransactionManager(Function<Configuration, TransactionManager>) when you want to set it as the default TransactionManager for all event processors.
Quite a mouthful, but I think this should help you further, @joey-ihomer.
Apart from that, I am curious why you and the team decided not to maintain your events in a store.
If you are at leisure to share, please do!
Apart from that, I am curious why you and the team decided not to maintain your events in a store.
If you are at leisure to share, please do!
Yes, @joey-ihomer even enlisted me to answer this.
Our application is managing connected IoT devices. A long time ago, people decided to use Axon. There is a Device aggregate in the ssytem and every time these devices send some data to us, this data is put in an Axon command sent to this aggregate. The aggregate then emits it in an event.
When this application was deployed, people operationally managing eventually asked why it was using so much storage. Well, it was the event store. The application was, at that point, not running in a mission-critical role and people weren’t willing to pay for ever-increasing storage for the unimportant service at the fringes of their cloud services. So people looked for, and found, a way to run the Axon application without the event store.
Skip forward another few years and the application is in more serious use. We find ourselves caring about consistency and resilience as evidenced by this question. We are using Axon, a framework that should be able to help us achieve these things, but we find that with the way the application is currently set up, we’re not reaping the benefits of Axon.
Some general observations that I have here:
Operational people really seem to hate ever-growing databases. Event sourcing literature tells you that you shouldn’t care because storage becomes cheaper faster than your data becomes bigger. But your average Joe the cloud engineer is not reading event sourcing literature and he is concerned about running out of disk space, or the size of a dataset hurting performance. Every event sourced application I’ve seen in production was anxiety-inducing for people in such roles.
We probably applied event sourcing to the wrong problem in our business at the beginning. Dealing with IoT devices is very event-driven, but the events are very many and not always relevant for the long term. I’ve come to think that event sourcing is a good choice for the deliberate human decisions with long-lasting impact: setting a price for using the device, attaching the device to a customer account, etcetera. But the automated notifications from the devices themselves, things like five-minute sampls of what their internal temperature is or something, that’s not something that I want to keep around in expensive highly-reliable high-performance storage for years and years. I wouldn’t use Axon for that.
But as always, ideas about how it would have been done better in the past are cheap. Now we have a system that is running 24/7 and that customers depend on. We’re trying to move towards the system we think we should have built in the first place, but it’s going inch by inch as we have to keep serving our current users all the time.
As you can see I find our case pretty interesting as well. If you’re also still interested I wouldn’t mind giving a talk about it on an AxonIQ Conference or so. It’s away from the typical “we had a legacy mess and then we used Axon to fix it” narrative and more of “we have a legacy mess and Axon is part of it - what now?”.
Most definitely true.
This is part teaching them the event sourcing literature, and part supporting them and saying the data has its benefits until they’ve read the literature.
Granted, there are caveats, as you highlight on in your second point:
In all fairness, I think you’re right. Using commands, events, and queries for communication is most definitely not the issue. Having a Device aggregate that (I assume) lives forever publishing “technical” events is likely not the way to go.
However, how to do it better, is hard to say for me without diving deeper in your domain.
Perhaps I am entirely wrong and there’s merit in setting it up as it is.
That’s awesome in its own accord! Congrats with that.
If you and @joey-ihomer ever need more fixed support or guidance on how to adjust your current application, know that that’s something AxonIQ and its partners facilitate.
Sadly enough, the CFP already closed for this year. Nonetheless, if you’d be up to proposing this for next year, I would definitely do so. These kinds of gotchas/lessons should proof helpful for a lot of people.
