And if we have not said it before, thanks for the awesome work on the Axon Framework : ) …
A thing that has bothered us for some time though, is how to handle re delivery of events to our event listeners from Rabbit. One such case is for example when a batch of events is fetched from Rabbit, handled in an event listener, persisted and committed to a database but Rabbit is down when they are about to get acked. When Rabbit gets back up the complete batch will be redelivered. One solution would be to use distributed transactions between the database and Rabbit, but that is a road we are a bit reluctant to go down…
Another solution is to make every event listener idempotent for them selves but wouldn’t it be even nicer if the infrastructure handled that for you?
This is where the question in the topic comes in. We would like to hook in a generic deduplication mechanism that for every cluster keeps track of which events has been handled and committed, preferably in the same transaction and data source as where the event handler stores the rest of its state and the best place to hook in such a feature that we find was a custom ClusterMessageListener. the DeDeplucatingClusterMessageListener would keep track handled events, silently ignore the once already handled and publishing the others to the cluster.
Do you have any input on the above reasoning and would it be possible to extend the ListenerContainerLifecycleManager with the possibility to configure a custom message listener, for example by fetching a prototype from the application context?
I am glad you're happy with Axon so far. But I'm not done yet. Things will only get better....
The acks sent to rabbit and the database commit are typically handled by the same transaction manager, if you use Spring. It's not 100% fail safe but the chance of duplicate messages seems to be very small. 2pc is not a full-proof solution either.
A relatively safe way would be to ensure the connection to Rabbit is alive just prior to the commit. If it is, commit the database and then Rabbit right away. There will be only a very little window of time where duplicate messages could occur.
However, if that small chance is still to risky for your specific usecase, deduplication could be an option. I'd implement it a bit differently, though. Using the same datasource is smart, because it allows for the atomicity that you require.
It would probably be easiest to implement a Cluster that keeps an administration of event id's that it has processed and forwards to a delegate cluster if the event is new. If it detects a duplicate, it will simply ignore the event. Since it uses the same data source as the event listeners, all should be committed in a single transaction.
Note that you should weigh the cost of maintaining the deduplication administration against the benefits of that littlebit of extra certainty. If idempotency is an option, always go for that one.
Your answers really shed some valuable light on the problem…
To start of, if things will only get better I’m looking very much forward to the future.
I like the simpleness of you poor mans 2pc (only checking for the liveness of the Rabbit connection before committing the JPA transaction) very much, and it will probable take us long way. Currently we are wiring up the ListenerContainerLifecycleManager with a plain JpaTransactionManager so it it is not involved with the Rabbit at all. Works very well under the assumption that the Rabbit is always available : ). But I guess we would have to implement this TransactionManager ourselves because I can’t find any such implementations that in any way would combine the two? If you’ll excuse a newb question…
Thanks for pointing us the direction of the Cluster, that’s of course where it should go. All we had to to was to look one step further…
About the last point, this is where I’m a bit confused. I realize that idempotency is the key here, but in many cases I don’t see how it can be implemented in any other way keeping track of all the events that you have received. For all those cases, letting the infrastructure handle it instead of adding the extra information needed to every table that has the need seems like a good option. Is this where my inexperience designing for these types of systems fail me or could there be something to it?
you can configure a transaction manager on the ListenerContainer(LyfecycleManager). And that should work with the JPA TransactionManager as well. I believe Spring will “attach” the Rabbit transaction to that transaction manager and send an ACK to Rabbit when the transaction is committed. I’m not sure it does a “is the connection live”-check, though.
I’m using that configuration in one of my own projects, and successfully executed some “crash the thing and start it back up” tests. Things went on where it left off correctly. But there is always a small window of opportunity for Murphy to kick in, whatever you do.
If you use a JpaTransactionManager Spring will hook in TransactionSynchronization which acks the messages after a successful commit yes. However, if your event listeners mostly insert rows to a database, most of the time will be spent during the commit and after that, there is no ‘is the connection live’-check. According to both my understanding and my tests, this opens a relatively big window for the rabbit to go down, of course depending on you txSize configuration. The bigger the txSize, the bigger the window, and with a txSize of 500 I managed to shut down the rabbit in the middle of a commit 100% of my tests : ) …
As we found no good way of registering new TransactionSynchronizations while using the SimpleMessageListenerContainer we ended up extending the JpaTransactionManager and doing a
transaction.flush();
rabbit.ping();
before the commit. After that, I have not managed to get any duplicates delivered by killing the rabbit during load.