Kafka a perfect fit for Axon ?

Imported from Google Groups
1

Hi guys,

We’re currently in early analysis phase of creating a new architecture using CQRS ES. We’ve been using Axon 2 CQRS ES for a couple of years on a legacy monolithic application (Oracle, …) and we want to rewrite it in a cloud native micro services way with more modern components.

My understanding of CQRS ES’s key architectural difficulty is that all events concerning a given aggregation root must be handled sequentially. I actually find this constraint to be more of a strength for it avoids all concurrency headaches and I would like to put this per aggregration root sequential way of thinking at the core of our architecture.

For it to be simple and efficient in a distributed environment, it requires a solid distributed event bus at the center of the infrastructure. One key constraint of that message broker is that it must support guaranteed delivery and cluster wide sequential processing of messages of a given message group (ie the aggregation root for CQRS ES). Messages being delivered more than once can be dealt with idempotent consumers.

We initially planned on using ActiveMQ’s JMSXGroupID (set to aggregation root ID) to achieve cluster wide per aggregation root sequential processing of events but despite having used ActiveMQ for many years, I still find it difficult to handle it production (weird unexplained crashes, memory tuning problems, cluster wide JMSXGroupID singleton consuming sometimes not being honored, …). Please, don’t take this as trolling, this is just my experience and I’m not feeling confortable in building all our critical infrastructure on ActiveMQ.

Then a guy in our team brought Kafka to the table and I love the simplicity of the shard based rooting algorithm, simple (at least to understand), efficient and amazingly performant.

Kafka seems to me like a perfect fit for Axon. I’ve read here and there about using Kafka for CQRS ES, there’s even a 2 years old integration from Viadeo and I’ve seen mentioned that Kafka support was planned for Axon 3 but I don’t see any mention of Kafka in Axon 3’s source code.

I therefore would like to know how close the Axon community is to being able to use Kafka as an event store. Is the Axon community and CQRS ES community at large confident that Kafka is the right tool for the job and if so, how difficult the integration would be and are there any known technical difficulties to overcome or is it simply a matter of finding the time to develop it ?

I’m trying to gather evidence and insight of the problem at hand before trying to integrate it ourselves.

Thanx for your time,

Cédric Vidal
Quicksign CTO
http://www.quicksign.com/

2

Hi Cedric,

we have been investigating Kafka to include support for it in upcoming Axon versions. Due to priorities, this hasn’t been planned for a specific version yet. The reason we mentioned Axon 3 for Kafka, is that Axon 3 will have a different design when it comes to Event Bus versus Event Store. Unlike Axon 2, you will only need one or the other. When using Event Sourcing, your Event Store will also act as the Event Bus.

My impression is that it will fit perfectly as an Event Bus. It might also fit as an Event Store, but my impression is that Kafka is not designed to keep messages stored forever, although apparantly, it can.

Cheers,

Allard

PS. If there is anything I can do to help implement Kafka support, let me know.

3

FYI

I’m currently scanning the Internet for best practices with CQRS/ES to put in use.
And I came across this part of Greg Young’s talk on ES (from 38:38 until around 41:54 ~ only 3:20 in total)

https://youtu.be/8JKjvY4etTY?t=38m38s

I will not try to beat explanation in the video, but I’ll summarize for those who what to read the point…

He distinguishes two types of subscription models:

  • Producer-driven (e.g. RabbitMQ)
  • Consumer-driven (e.g. Kafka)

The problem with Producer-driven subscription system is that it requires additional “control channel” for a consumer to the producer.
Basically, every time a consumer needs data from an event store, it has to use “control channel” to instruct producer to prepare a queue and events in it for this consumer.

The opposite approach is Consumer-driven subscription system where the position in the stream is remembered by the consumer itself.
In this case, the consumer simply positions its own “iterator” anywhere in the stream (the “iterator” is called “offset” in Kafka) and read events sequentially.

Obviously, the absence of “control channel” in the second case is not true - for the consumer-driven subscription the “control channel” is simply elegant built-in “offset”/“iterator” thing.

There are few reasons I’m “enriching” this thread by my post:

  • (Correct me if I’m wrong) It makes a perfect additional clarification for “Event Bus versus Event Store” words by Allard. Indeed, if Kafka is an Event Store with its elegant Consumer-driven subscription approach Event Bus becomes redundant as the customer will use private “offset” to consume events.
  • And I also remember mailing lists topics which went something like this “How do I replay both historical events from t(0) to t(then) and then all events happened since t(then) until t(now) seamlessly?” - this is what also discussed in the video piece above. Kafka is the natural solution here.
  • It also seems like Kafka is the perfect out-of-the-box “Record and Replay” server I’ve been thinking about (see “Record and Replay – realtime playback of system state” thread - https://groups.google.com/d/msg/axonframework/KDbdy4MN8bA/Gx7n4n7gBgAJ). Obviously, I’m after support for Kafka.
4

Hi Alexey,

The headline feature for Axon 3 is a new consumer driven approach to event handling. It works just like you describe, but is compatible with any event store (current implementations: JPA, JDBC and Mongo).

Event consumers will be able to stream events in the store from a given position and store this position. The new approach will also make it possible to distribute events without having to rely on publishing queues like RabbitMQ (though this is planned for after the upcoming 3.0 release).

Obviously these new APIs are also a perfect fit for an event bus/store implementation that uses Kafka, but work on this hasn’t started yet.

Regards,
Rene

5

Hi Rene,

Thanks for the info!

I finally reached this open tab with SO question in my browser:

http://stackoverflow.com/q/17708489/441652

Nothing beats the practice, but both answers suggest that Kafka has no issues with being used as an event store with unlimited retention.

6

Has couchdb been considered as an event store for axonframework? It has a changes feed [1] that allows consumers to provide a parameter “since” that is the id of the last set of changes that were read. This is similar to the kafka offset. Couchdb 2.0 will be getting released soon which has clustering allowing it to scale to much larger database sizes [2].

[1] http://docs.couchdb.org/en/stable/api/database/changes.html
[2] https://blog.couchdb.org/2016/07/25/the-road-to-couchdb-2-0/

7

Thanks for the pointer. Definitely looks interesting enough to do a PoC with…

8

Great stuff. I’ll ping you offline with my contact details, I work with IBM cloudant and we have a close relationship and committers on the couchdb project so should be able to help with questions.

9

Allard,
I want to make sure I am understanding this correctly. If Axon 3 uses the Event Store as an Event Bus, is that a feature or a requirement? in that can they be run separately as they are in Axon 2? then Kafka could be used for the Event Bus, but not the Event Store. Otherwise the only way us use Kafka in Axon 3 (with Event Sourcing) - is as the Event Store… please let me know if this is correct, or if I am missing something

thanks
jason

10

Hi Jason,

in this case, I’d say it a feature, but you can still use both Kafka and an Event Store, without Kafka being the Event Store. You could define an Event Processor that uses Kafka as its source of Events, instead of the Event Bus. You would then have to configure a component that subscribes to the Event Bus (or Store for that matter) and forwards all events to Kafka. In that case, you’ll have a scenario very similar to that in Axon 2, where your Event publication mechanism is different from the storage, except that you’re not forced to separate, and you could even have more than one implementation for dispatching Events.

Note that the EventProcessor configuration API might change a bit when we’re going to implement AMQP. I expect that the SubscribingEventProcessor will no longer be strictly tied to an EventBus, but from a Subscribable Source of Events, of which the Event Bus is an example. But an AMQP message broker or Kafka stream could also be sources.

Cheers,

Allard

11

Hi here is good suggestion about event bus https://youtu.be/8JKjvY4etTY?t=2266

12

This is exactly how the Axon Event Store works (which also implements Event Bus).
It supports both models of receiving messages (push/pull).

13

Hello guys,

For the record, I just wanted to report that we’ve successfully gone with Kafka Streams to implement CQRS-ES. Following Confluent CTO Neha Narkhede’s blog post of Sept 2016 Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?
https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/

Kind regards,

Cédric Vidal
Quicksign CTO
http://www.quicksign.com/