Kafka Message Keys and Axon's EventBus

Imported from Google Groups
#1

Hello all, another question as it relates to Kafka our message broker.
Kafka utilizes a concept known as partitions in order to improve throughput of message consumption. This is caveated with the fact that each partition has a unique set of messages and when parallized against multiple application instances we risk messages being consumed out of order. The solution to this is to provide a message key to events where order matters.

I do not see an obvious way to do this for events produced through an Aggregate’s apply() or eventBus.publish().

A workaround might be to intercept all outgoing events and decorate them with the appropriate key. I need advice on how to best do this.
The second part of this approach is determining which key to use and the current thinking is to pass a value into MetaData and then read this value out and set it as the key.

I hope this is not too difficult and does not result in a significant performance impact.

#2

Hello Michael,

Kafka is more an event log then a broker so to speak. It is designed with the ddd philosophy, so when you send an event it should always (almost all the time :-)) be related to an entity.

Kafka uses a key to shard the events into partitions, so if you use entity id as the key, your events will always be consumed in order from entity perspective.

Moreover, it can also snapshot your entities states, keeping the last event for each of your entities (be careful with this behaviour).

FistName: Frédéric
LastName: Gendebien
Mobile: 0493132433
Email: frederic.gendebien@gmail.com

#3

Hello Michael, Frédéric,

Axon uses the concept of a SequencingPolicy to decide in what batch an event would fall under when it comes to Event Handling (to update query models for example).
This exact same SequencingPolicy can be configured for the DefaultKafkaMessageConverter in the Kafka Extension, which is used to define the key of a Kafka Producer Record.

Thus, you will have to perform zero effort upon the publishing part of events which, if you ask me, is highly desirable too.

The framework provides a couple of implementations for the SequencingPolicy, but you are obviously free to create your own custom version which applies to your Kafka set up.

Hope this clarifies things for you.

Cheers,

Steven van Beelen

Axon Framework Lead Developer

AxonIQ
Axon in Action Award 2019 - Nominate your project