'Outbox' pattern and using streaming processor

Axon Framework
1

Hello,

My team is developing microservices using Spring Boot and Rabbit MQ. In some of our services, we use the Axon framework (without the Axon server). We publish events from such services via axon.amqp.exchange=ExchangeName.

We have a main order-service where we use Axon, and we critically need to have atomicity in writing the order event to the database, and publishing the event to the RabbitMQ. If the connection to RabbitMQ, the connection to the database, or the instance is restarted (or all at once) during order processing, we cannot allow the event from the order-service to get lost and not get to RabbitMQ, but persist in the database.

I studied a little the articles that are present here on the topic of the Outbox template, but I did not see such an option anywhere.

  1. Still, I stopped at the fact that axon.amqp cannot provide 100% event delivery to the RabbitMQ when services/connections fail, am I right?
  2. What if I use a streaming processor and an event handler as an proxy for publishing events to the rabbit manually? That is, in this case, won’t I implement the ‘outbox’ pattern, due to the fact that the token entry table will be used as the same ‘outbox’ table?
  3. In the axon documentation, I did not see any specific recommendations or restrictions on what exactly the event handler can do except write to the database? (from a performance point of view). That is, for example, publishing an event to the rabbit? Sending an HTTP request to another service? How bad is it to do this?
2

Hi @vladkuzma993, and welcome to the forum! :wave:

Let me go through your questions one by one:

Answer to question 1

To be extremely specific, I am wondering if there is any form of application that can provide a 100% delivery guarantee :wink:
Joking aside, if I understand your questions correctly, you want to ensure that if sending the event over AMQP fails, that the event is still stored in the Event Store to be processed at a later stage?

If that’s a correct assumption from my part, then yes, it’s tough to set this up with your current infra. If you are NOT using Axon Server, I am pretty sure you’re using a RDBMS to store events in (another assumption, is it correct?).

When using Axon’s AMQP extension, the SpringAMQPPublisher is in charge of publishing events. This SpringAMQPPublisher uses a SubscribableMessageSource from Axon Framework to receive its events. Axon provides several implementations of this SubscribableMessageSource, being:

  1. The AbstractEventBus - local abstract installment of the EventBus, used by Axon Framework for sending messages within an application. Publishing of events occurs in the same thread as the one invoking your command handlers in your setup.
  2. The SpringAMQPMessageSource - Subscribable source based on AMQP. Any characteristics of dealing with failures fall unto AMQP in this example. Pretty sure this will not be a valid option in your scenario, as you’re talking about events coming from an Axon application.
  3. The SubscribableKafkaMessageSource - Subscribable source based on Kafka. Kafka’s retention period makes it so retrying becomes doable for sure. However, same as with AMQP, I don’t think this is a source you can use for the same reason why AMQP isn’t an option.
  4. The Persistent Stream - This is a managed source by Axon Server, leaving retry behavior with Axon Server entirely.

Given what I am assuming about your infrastructure, I am pretty sure you’re in scenario 1. The downside here is, that the thread that handlers the command and publishes the event, is the same one that invokes the SpringAMQPPublisher to publish to AMQP. Hence, if your AMQP setup fails due to an exception or network failure, the exception will bubble up and cause the entire command handling operation to fail.

Result: your event does not get stored. And thus, cannot be retried for publishing over AMQP. If this process would be separated from one another, through, for example, the out box pattern, as you point out, you could proceed.

Or, you have a setup that (1) handlers the command and stores the event and (2) provides the event to the SubscribableMessageSource for the SpringAMQPPublisher. You could achieve this by using options 3 (kafka) or 4 (persistent streams from Axon Server).

Answer to question 2

This would be a way forward for sure. You could argue if it’s really the outbox pattern, but I do think the suggested flow would help! It, essentially, aligns with my previous suggestion to separate the process from handling the command and publishing/storing the event, from putting the event on AMQP.

Answer to question 3

I don’t think there is inherently anything bad with calling another service like this from an event handler, @vladkuzma993! If the operation is “reasonably fast” it is fair game to perform in an event handler. With reasonable fast, I’d sit somewhere between 50ms and 1s (these are pure gut feeling numbers by the way!).

However, it is hard to deduce what is feasible for your application. How fast something is, is really a non-functional requirement of your application. If sending to AMQP is very important (which I guess, given that you’ve written a post about it :wink: ), then having a dedicated StreamingEventProcessor just take that effort seems entirely feasible to me. Grouping other event handling tasks under the same StreamingEventProcessor would impact the overall process somewhat, of course.

Concluding, I would measure the time it takes to publish an event from a StreamingEventProcessor in “Axon-land” to AMQP. If it’s within reasonable bounds for your application, then you’re good to go. If anything, Axon Framework is not very picky on what you do in an event handler :slight_smile: