Kafka EventStore for Axon 3

Imported from Google Groups
#1

Hi Folks,

I’m very excited about the features of Axon Framework and we want to use it for implementation of CQRS ES in our application.
I read a lot about different impplementation of event buses (including the discussion about the eventstore/eventbus and producer/consumer based messaging).

As far as I understand Kafka Messaging could be a perfect implementation for an event store (which means both: event distribution and storage) and would allow to replay events, if the query side intends to do so.

I found an implementation of a Kafka Terminal for Axon 2 (https://github.com/viadeo/axon-kafka-terminal) which seems to implement messaging only.

I would like to implement the Kafka Event store and looking for:

  • Any feedback if this is a good idea
  • Any progress someone made
  • Any hints regarding the implementation -> what to look at and what to build on
  • Any details, doubts or reports of someone who already tried that.

I think it is also a good idea to wait for Kafka 0.11 to be released (probably in few days), because it will support Record Header (https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers).

Thanks for your feedback in advance

Kind regards,

Simon

#2

Hi Simon,

I haven’t looked into it in detail myself, but people that did, told me that Kafka is not a good match for an Event Store. Apparently, the storage model of Kafka isn’t suitable for having both a large stream of sequential events (which it can do just fine) as well as being able to load events based on an AggregateId.

You might want to look into this in more detail, before attempting a full-blown implementation.
Please keep me informed about your progress, and if there is anything I can do to help, let me know.

Cheers,

Allard

#3

Hi Allard,

I noticed, that there are several problems here.

First one is the TokenStore. My naive idea was to use the Kafka consumer offset to identify it, but I’m not getting it working yet.
-> I’ll try a little different approach here and will inform you.

The second problem is that Kafka thinks in infinite streams and the EventStore is not. So at some point of time there must be something like a buffer to convert an infinite Kafka stream to a DomainMessage stream expected by the EventStore. This can become a problem if the EventStore is replaying the last 70K events… I’ll think about it.

The third problem is that the EventStore is looking for events for AggregateId and it does it in some situations. That means in Kafka terms something like, move the offset to the beginning of the topic and give me all records, but throw away all not matching - not nice at all… Kafka doesn’t have any query semantics, so, probably we could use Topic-Key for this purpose…

Stay tuned…

Simon

#4

Hi Simon,

Currently, I’m working on POC for Axon Framework 3.3. We want to use it for ES & CQRS concept. And also I need to use event store as Kafka/kafka Streams here. I read lot of tutorials regarding axon-kafka & kafka streams but couldn’t find matching samples for this scenario. Please let me know if you have any samples which is used event store as Kafka with Axon framework.

Thanks in advance.

#5

Hi Simon,

we don’t have any samples for Kafka as an event store. The main reason being that Kafka is not suitable for that purpose. Kafka is a store-and-forward message broker, and lacks some of the features that are required to reliably use it for event sourcing. Most posts out there from people that say they do event sourcing with Kafka actually don’t event source, but do event streaming (or in some cases have a lot of accidental complexity to compensate the missing features in Kafka). Both happen in event-driven architectures (sometimes at the same time), but are significantly different.

This doesn’t mean there is no place for Kafka in a system that uses Event Sourcing. Kafka is very suitable for so-called external events. Those are the events that get exposed beyond the bounded context of the service that generated them. Typically there is filtering and transformation involved before internal events ( those originating from event-sourcing) are published for consumption by external services.

I hope this makes sense.
Kind regards,

Allard Buijze

twitter-icon_128x128.png

#6

We’re using Kafka as a message broker.
Using it as an event store (we’re using Postgres) is ill-advised for the following two reasons:

  • retention of -1 (you’ll be paying a lot in costs)
  • There is keying but there isn’t really any way to “index” within kafka. So replaying events for an agggregate by id from kafka each time is highly ineffective (especially over time)

The approach we decided to go with was having kafka seed new services and populate the event store when we want version the entire service.

#7

Hi Allard,

As you suggested, can we do with kafka streams(amq streams) as event store with Axon framework? What is your opinion?

#8

Hi Erandika,

in short, I would just say you really can’t.
You can probably make something work for the very straightforward happy-path scenarios, but as soon as there is a bit of load on the system, the chance of running into concurrency problems is very significant. Kafka simply doesn’t validate incoming messages against any of the already published ones. Something that an Event Store should be able to do, in order to be able to rely on the correctness of the event stream.

Cheers,

twitter-icon_128x128.png

#9

Hi Allard,

Okay, Got it. Thank you very much for your kind information. & I will get back to you if I have any queries on the axon framework.

Best Regards,
Erandika