Reading from two Kafka topics in an ACL

Imported from Google Groups
1

Hi everyone,

So my team is creating a proof-of-concept suite of microservices using Axon Framework. When we started v3.3.5 was the latest version so that’s what we’re using.

First, a little description. We have two bounded contexts - let’s call them “reporting” and “verification.” Each context contains one microservice at the moment. Both microservices are running on top of Spring Boot. Since these two microservices need to talk to each other we created an anti-corruption layer (ACL) between them. Each microservice receives commands via Spring Cloud, and each microservice dispatches events via Kafka. The ACL subscribes to the Kafka topic and translates events from one microservice into commands for the other. The ACL uses a Saga class to handle the events. So far this is working just great.

However, we don’t really want both microservices putting their events in the same Kafka topic. We’d like each of them to have their own topic. After searching through this Google Group and reading through the source code a bit it seems we can accomplish this using two Tracking Event Processors - one for each topic. So, my first question is - is that correct?

Assuming it is I imagine you’d somehow associate the existing Saga to both Tracking Event Processors? How would you go about configuring these Tracking Event Processors with Spring Boot?

Appreciate the help!

2

Since I don’t seem to be getting any hits let me try and attack this from another angle.

Is there any documentation on using the Configuration API to configure a tracking event processor anywhere?

3

Hi Michael,

Understandable that you bumped the question you’re having, as things might not be overly clear in regards to this topic.
So, given the scenario you’re in, I think I can give some background on the situation.

What you’re essentially looking for is to have a single Event Handling Component read from two distinct Event Streams at the same time.
This translates to your scenario in two different Kafka topics, which you want a given Saga type to receive events from.

It is correct that to utilize the Kafka Extension currently maintained by AxonIQ to be the source or events, you’ll have to configure a Tracking Event Processor.
However, it is a little more intricate then that.
A Tracking Event Processor requires an interface called a StreamableMessageSource to be able to retrieve an Event Stream and delegate it’s contents to Event Handling Components (like a Saga).
Implementations of a StreamableMessageSource are, for example, the EmbeddedEventStore, but also the KafkaMessageSource.

Both implementations however only deal with a single source.
Thus, they do not give you the option to merge several Event Streams in to one, as I believe I tried to explain to you on this GitHub issue as well.

By the way, my sincere apologies if that didn’t come across thoroughly enough, which I am assuming as of this User Group request.

On top of the StreamableMessageSource explanation, I feel it’s important to emphasize that you cannot set up two (Tracking) Event Processor for a single Event Handling Component, like a Saga.
If you would configure this (by the way, for the configuration, click here), this would essentially mean you’d duplicate the number of Saga instances, without merging the distinct Message Streams in to one.

The way forward with this desired set up, would be to introduce a StreamableMessageSource implementation that can deal with N number of sources.

This is a feature we are working on at AxonIQ and can be found as issue #1101.

The short answer to your problem at the moment, Michael, is that you will have to use the same topic to be able to get all the events in the expected order at your Saga type.
Once issue #1101 is resolved however, it will be possible to merge several sources into one and provide that as the means for a Tracking Event Processor to stream events.

Hope this sheds some light on your situation, and let us know if you have any other question.

Cheers,
Steven

4

Thank you very much for your reply, Steven. You’re correct that I didn’t quite understand you when you replied to my GitHub issue. I do now, though, and I thank you for your patience.

I may be able to assist you on issue #1101 if you need it. It would be in my best interest to do so as I really need that feature for my work. I fear I may end up slowing you down, though, due to my unfamiliarity with the code base. Let me know either way!

5

Hi Michael,

Your effort in regards to #1101 has been noticed, thanks for that!
FYI, Allard is currently resolving the feature, so I am pretty confident it’s gonna make it to Axon Framework release 4.2.

Cheers,
Steven