We’re currently in early analysis phase of creating a new architecture using CQRS ES. We’ve been using Axon 2 CQRS ES for a couple of years on a legacy monolithic application (Oracle, …) and we want to rewrite it in a cloud native micro services way with more modern components.
My understanding of CQRS ES’s key architectural difficulty is that all events concerning a given aggregation root must be handled sequentially. I actually find this constraint to be more of a strength for it avoids all concurrency headaches and I would like to put this per aggregration root sequential way of thinking at the core of our architecture.
For it to be simple and efficient in a distributed environment, it requires a solid distributed event bus at the center of the infrastructure. One key constraint of that message broker is that it must support guaranteed delivery and cluster wide sequential processing of messages of a given message group (ie the aggregation root for CQRS ES). Messages being delivered more than once can be dealt with idempotent consumers.
We initially planned on using ActiveMQ’s JMSXGroupID (set to aggregation root ID) to achieve cluster wide per aggregation root sequential processing of events but despite having used ActiveMQ for many years, I still find it difficult to handle it production (weird unexplained crashes, memory tuning problems, cluster wide JMSXGroupID singleton consuming sometimes not being honored, …). Please, don’t take this as trolling, this is just my experience and I’m not feeling confortable in building all our critical infrastructure on ActiveMQ.
Then a guy in our team brought Kafka to the table and I love the simplicity of the shard based rooting algorithm, simple (at least to understand), efficient and amazingly performant.
Kafka seems to me like a perfect fit for Axon. I’ve read here and there about using Kafka for CQRS ES, there’s even a 2 years old integration from Viadeo and I’ve seen mentioned that Kafka support was planned for Axon 3 but I don’t see any mention of Kafka in Axon 3’s source code.
I therefore would like to know how close the Axon community is to being able to use Kafka as an event store. Is the Axon community and CQRS ES community at large confident that Kafka is the right tool for the job and if so, how difficult the integration would be and are there any known technical difficulties to overcome or is it simply a matter of finding the time to develop it ?
I’m trying to gather evidence and insight of the problem at hand before trying to integrate it ourselves.
Thanx for your time,