How do we guarantee only-once delivery with Axon AMQP and RabbitMQ?

Axon Framework
, ,
1

Hi, I’m running into an issue of reliability with the delivery to EventHandlers through RabbitMQ. It happened recently that there was an unexpected shutdown on one of our systems, and this led to 5 items on the queue being redelivered, even though they were already handled.

Our stack is multiple Spring Boot applications, set up as a microservices architecture, and a separate self-hosted RabbitMQ container to allow communication between microservices.
For this we use a subscribing Event Processor with very basic configuration. We use the axon-amqp extension for this. Every microservice has a separate queue and events are published to all queues. Has been running in production for a few years and no complaints :+1:, except for this question we ran into now. :slight_smile:

We tried to look into using a Streaming Event Processor instead of subscribing, since it is said to make it more resilient to unintended shutdowns. However, the axon-amqp extension does not seem to have support for this. (there is no implementation of StreamableMessageSource in the axon-amqp library, and the reference documentation also only mentions a subscribing processor).

Are there improvements we can make to guarantee only-once delivery when using RabbitMQ/AMQP?

2

We indeed don’t have support for RabbitMQ as a StreamableMessageSource, @Wmaarts. Although my recent RabbitMQ knowledge is a little rusty, I do still think it does not provide a means to request opening a message stream from a given position in said stream, correct?

If through whatever mechanism this is possible (so, the queue is persistent, at least partially (as with Kafka)), then we could make this work.

Point aside, let me move to your main question in this topic:

To be frank, I think it would be easier to make the event handlers in question idempotent then making the infrastructure layer guarantee only-once delivery. Even if you could move to a StreamingEventProcessor setup with your Axon Framework applications, you could still have the scenario that the TrackingToken (read: position in the stream) has been dealt with weirdly, causing some events to be handled twice.

If your event handlers are idempotent, that would simply not be a problem anymore.

Then again, perhaps it’s not feasible to make them so. If they aren’t, could you perhaps elaborate what the event handlers in question do that accidentally had 5 items handled again?

3

To answer the first part: I’m no expert either but I could find some documentation about rabbitmq supporting streams with a plugin. Streams and Super Streams (Partitioned Streams) | RabbitMQ and Stream Plugin | RabbitMQ.
But we don’t use this at the moment.

Making the event handlers idempotent is definitely an idea worth looking at, thanks. We have this in place by accident in some places, i.e. where a state is updated in some way, updating it twice to the same state isn’t a problem.
We’d have to look case-by-case for our EventHandlers but this might be feasible.

In the case of the 5 events that reappeared: simplified, the eventhandler sends a command that adds (financial) transactions to an aggregate. So these 5 transactions were duplicated, because we generate a random id for each transaction.
We could make this idempotent by adding a check to see if this exact transaction already exists before we generate a unique ID. That would solve my immediate problem. :white_check_mark:

… that’s just one eventhandler though. We have about 60 cases in our codebase where Events publish new Commands. We might have to check them all for idempotence if there is no guaranteed way to solve this in the infrastructure layer. :thinking:

4

To wrap this up: we did not find a way to guarantee only-once delivery with RabbitMQ on an infrastructure layer.

Instead our answer was to make our EventHandlers (more) idempotent.

1 Like
5

That’s a great find @Wmaarts! Is it perhaps worth sharing so that others reading this thread know where to go?

I assume you will not be working on this idea further, right? Simply because the “60 cases” you’d have to check.

6

I’m assuming you misread, because we did not find a solution at the infrastructure layer. :sweat_smile:
So we’re going to make our EventHandlers more idempotent, that was our solution so I marked it as such for this thread.

7

Ow darn, my apologies. :bowing_man: I most definitely misread your reply, @Wmaarts.

Regardless, if there are any follow-up questions you might encounter in making your event handlers idempotent, be sure to shoot a message to the forum again!

1 Like