Use of axon kafka connector along with event store and tracking event processor

[Ashwini_Kumar]

Hi,

As per this discussion in the below mentioned axon google group

Tracking Event Processor for Event Replay Reading from Event Store.

Hello, I have a successfull implementation of the events replay using the tracking event processor. The PROBLEM is, it seems like the tracking event processor is reading events directly from the events store rather than the event bus. So, any time the command service pushes a new event (To Event Store and Event Bus) the event handler on query side is invoked and it consumes the event from the store rather then the bus. My Application Config App: Spring Boot Event Bus: Rabbit MQ Event Stor…

I understand that the tracking event processor using the event store for processing of events then what is the use of axon kafka connector if I am using kafka as my event bus if the processing of events is dependent on event store. Can I remove my axon kafka connector in that case and will my application work as expected?

I am using axon kafka connector along with tracking event processor so how can I ensure that my tracking event processor always reads the events from kafka topic rather than from the event store?

[Steven_van_Beelen]

Hi @Ashwini_Kumar,

You could indeed go for either option when it comes to event streaming. However, I am assuming you are using Axon for Event Sourcing as well, correct? Kafka, or Axon’s Kafka Extension, just don’t support Event Sourcing in an efficient matter. Henceforth you wouldn’t be using Kafka as an Event Store solution, but simply as a broker. This by Allard explains nicely what you would need from an Event Store, but also why Kafka isn’t a good fit (definitely worth a watch).

However, you can very well use it for event streaming as pointed out. Henceforth Kafka and thus the Kafka Extension provide means to connect to Axon’s Event Processors like the TrackingEventProcessor (TEP). When it comes to configuring this, I highly recommend to check out our Reference Guide on the matter. More specifically you would need to use a StreamableMessageSource for a TEP in general, hence requiring you to configure the StreamableKafkaMessageSource as the event source of your TEP. The linked ref-guide section should show you how. On top of that, you can check out the sample which is part of the extension repository if you like, which also shows how to set this up (albeit in Kotlin).

So concluding, Kafka is fine for event streaming, not so much as an Event Store. If you are looking for a consolidated solution for both, I would recommend giving Axon Server a spin. It is a tailor made solution to be an Event Store (thus supporting both event streaming and sourcing), as well as a dedicated messaging platform. Hence you would no longer be required to configure Kafka to dispatch messages between connected application. The same would apply for commands and queries by the way.

Hope this helps you out further @Ashwini_Kumar!

Cheers,
Steven

[Ashwini_Kumar]

Thanks steven for your answer.
Just to make my understanding clear if I want to use kafka for event streaming then I will have to configure streamablekafkamessagesource as per the reference guide and then only if I will be able to use kafka for event streaming.
Currently I am using axon kafka connector with default configuration but I always see that my event handlers are always reading from event store rather than events published on axon kafka topic.

[Steven_van_Beelen]

You indeed need to consciously configure the StreamableKafkaMessageSource as the source to use for your TrackingEventProcoessor instances. By default, Axon will use the EventStore, which I guess is an RDBMS solution in your case.

As Kafka simply isn’t an EventStore implementation, it would never be the automatic default within Axon. Trusting this clarifies things further!