Multiple processing groups not able to consume the same kafka topic

Imported from Google Groups
#1

I have a microservice with the following configuration in my application.yaml:

![application.png|343x336](upload://hcTWvExcUUWoJs77bdOSORfqqZx.png)



































































When I annotate my Saga with the "sagaprocessor" processing group, and annotate an event handler with "myprocessor", whenever I receive an event in the default-topic, only one of them processes the event, instead of both.
If I remove one of these processors config in the application.yaml, and assign both the Saga and the Event Handler the same processing group - then it works as expected, both of them receive the event!
However, it is in my understanding that if both share the same processing group, then I wouldn't be able to replay the events for the projection because they would be replayed for the Saga as well.

What would the solution be? Is there a way to get two processing groups listen on the same topic or is there some other way to avoid conflict? Thanks!
#2

Hi Filip,

From Axon Framework’s perspective, the above configuration should work just fine.
What you have defined with the kafkaMessageSource is a StreamableMessageSource implementation.
This object can be used for any Tracking Event Processor, so that it can retrieve a stream of events using the Tracking Token to define from where the stream should start.

What you did with defining two distinct Processing Groups in the mode tracking, is define two different Tracking Event Processor.
One which services the sagaprocessor and the other which services the myprocessor processing group.
Both Event Processor use the same StreamableMessageSource, which is your kafkaMessageSource, but definitely to not permanently consume it.

My first hunch is, and this is definitely a hunch, that it might be a Kafka configuration thing?
My Kafka knowledge does not extend that far, but have you configured the topic to maintain the events?

Or will it drop/release them as soon as they’ve been read?

If it is the latter, then that would explain why a given event which you retrieve from the Kafka Message Source is only handled by one of your Event Handling Components.
If the issue still persist, I think we need more information to deduce what the problem is.

Things like the Axon Framework, Kafka Extension and Kafka version.
And how you have configured the message source for example.

Let’s try to figure this out Filip!

Cheers,
Steven

application.png

#3

Hi Filip,

I’m facing the same issue when trying to configure multiple tracking processors under same consumer group listening to the same topic.
Did you manage to identify the root cause ?

Regards,
Roy

#4

Hi all,

this is indeed one of the known issues currently in the milestone release, which is blocking it from going GA. We’re currently working on it, but need to juggle the time we have for this, with other requirements that we have.

Stay tuned…