I have a small Axon application in SpringBoot, which uses a JpaEventStore and an EmbeddedEventBus. No distribution here. It is a very simple use case, Messages from Kafka are converted into events and published to an EventBus directly, with an EventListener, that populates a JPA projection. I am not even using Aggregates.
My question is, is it possible to scale something like this horizontally? Scenario: Messages from Kafka arrive faster than the single EventHandler can process the resulting events. I would like to start more pods of that service and have multiple EventListeners process the events. But it seems it’s not that easy. If I start a second pod, it’s EventListener is not doing anything.
Any hints if and how something like that can be achieved?
When you say “more pods of that service” do you mean that each processes the Kafka messages? Or do you mean that you only have one that process the messages and then attempts to send events to multiple event handlers? If the later what do you use for the distributed event bus?
I don’t use a distributed EventBus at the moment, just the EmbeddedEventBus and a JpaEventStore, there is not much to distribute here. Kafka message processing is pretty fast, so Events get added to the store pretty fast as well. Processing the events might take a bit longer, depending on how complex the projection is, so it could happen that there is a lag in event processing. I wonder if I can scale the app and start more pods, so they can work on the events in parallel, working with the same token or something.
Is that something I can only achieve with a distributed EventBus?
I’m not 100% sure I understand your scenario and expectation, so correct me if I’m wrong in the following assumption.
There is one application that has two components:
one to receive messages from Kafka and produce events (stored in local event DB)
one to process the events
Now you start other applications with just the second component and connect them to the same local event DB. You expect they will pick up events from the event DB and process them in parallel, but they don’t?
If that’s the case, then I think you may need to manually configure event segments for each node. That said, it’s probably best to wait for some of the @AxonIQ engineers to confirm this or propose something better
Now you start other applications with just the second component and connect them to the same local event DB. You expect they will pick up events from the event DB and process them in parallel, but they don’t?
Yes, that’s basically it. I have read of segments before, but could not really figure out if that would help and what exactly I would have to do…
