Axon + Kafka Outbox

Hi,

I’m surveying persistence/architecture options for a new application and I’m not very knowledgeable about Axon yet. I hope you can help me with the following question.

I am interested in using Event Sourcing with either a relational database or MongoDB.
The application should write some events to Kafka – maybe a subset of the internal events or some summary events but not all internal events.
To write to Kafka, I want to use the Outbox pattern to achieve guaranteed delivery.
That is, the events for Kafka should be written to an “outbox” database table first and later be written to Kafka using Debezium.
See Reliable Microservices Data Exchange With the Outbox Pattern

Is that possible/sensible with Axon? How? Any comparable examples?
Could I control exactly what is written to the Outbox table?

I have seen that the Kafka extension exists.
It does not seem appropriate for my requirements.
An end user may perform an action that leads to “SomethingHappened”.
The write to Kafka may not be possible at the moment because of a network issue.
It does not make sense to wait for X seconds and then undo the transaction and interfere with the user flow.
But I still need to guarantee the Kafka event.
The outbox pattern is a classic approach for this.

Maybe this helps?

Thanks. It’s related.

In my case, I’m not interested in Axon providing that out of the box or with Axon Server.
I’m only interested in whether/how I could do that now with an RDBMS or MongoDB.

The guaranteed delivery is important for my scenario.
If Axon cannot achieve that, it would not be a good fit for me.

Hi Phillip,

Maybe you can try to write an EventHandlingComponent that performs the “selection and translation” step to filter out the right events to send. This component would perform the task you’ve described as such:

The application should write some events to Kafka – maybe a subset of the internal events or some summary events but not all internal events.
To write to Kafka, I want to use the Outbox pattern to achieve guaranteed delivery.
That is, the events for Kafka should be written to an “outbox” database table first and later be written to Kafka using Debezium.

For this component to receive the event-sourced events, it should be backed by a Tracking or PooledStreaming processor. Both are Event Streaming Processors available in Axon Framework:
the Tracking Event Processor (TEP for short), and
the Pooled Streaming Event Processor (PSEP for short).

You can find more information on these processors this section of our Reference Guide.

3 Likes