Axon in multi node setup using PSEP(Pooled Streaming Event Processor)

Axon Framework
,
1

Hello all,

We want to run axon in multi node environment. We have our setup on kubernetess. Our application has an auto scaling setup with Min 5 pods and Max 10 Pods. We would like to setup axon to be able to utilise all or max pods when our projections run in axon.

We think of using PSEP with 10 segments and maxSegmentCount of 1. Now in this setup what happens when the no. of PODS is 5 and segments are 10?

When the application scales up we would see the pods increasing to a count of 10, slowly utilising all the segments. Now when the pods begin to scale down from 10 to 9 to 8 and so on, how would the segments get reassigned? Would we have any loss of processing of data via axon?

How many segments and maxNoOfSegments would you recommend for our setup with 5-10pods? And what should be the Worker thread count: Should we consider the no of core counts on the machine?

2

Hi @Sneha_Bagri and welcome to the forum!

First and foremost, if you’re going to run your Axon Framework application with multiple instances, I would strongly recommend to look into distributed version of Axon’s Message Buses.
The easiest step to take here, is to start playing around with Axon Server actually.
It will give you a distributed command, event, and query bus, with all the required characteristics you would expect.

So, apart from that, we can move to the PSEP.
Firstly, I think it’s beneficial you’re selecting the PSEP for your environment as the Event Processor, as it provides more flexibility for scaling your instances.
However, if you set the maxClaimedSegments to one, you lose the flexibility.
Although by setting it to 1 you enforce each instance of your application receives a segment, you would increase your eventual consistency massively.

To achieve the flexibility you are looking into, it’s best to leave maxClaimedSegments to its default (Short.MAX_VALUE) and move to a setup where you consciously tell instances to scale those segments between one another. This is something you can, for example, set through Axon Server, by adjusting the load balancing strategy to a “per thread” setting.

In doing so, Axon Server will take up the role of “telling” different Axon Framework instances to release and claim. segments, until an equal distribution is achieved.
Besides Axon Server, AxonIQ’s working on a dashboard you can connect your application to, which will provide similar functionality (and more).

And if that also doesn’t work out…well then I recommend you checking the operations on the StreamingEventProcessor. It provides an API to split, merge, release, and claim segments, which are the exact operations Axon Server and the aforementioned dashboard uses.

Concluding, I hope this helps you further, @Sneha_Bagri!

3

Hi @Steven_van_Beelen, I’m interested in this dashboard you mentioned AxonIQ working on… Is there anywhere we can track the progress of this and the features it’s expected to have?

1 Like
4

Thanks for your interest in this dashboard, @Colin :slight_smile:

We currently do not have a public board for this dashboard.
Furthermore, as it’s an internal project, we’ve honestly not discussed whether we should make the board public to begin with.

However, there’s one public repository for this project, which is the Framework client (that would be attached to your project to “make things work”). If you’re curious, you can check that out here.

Apart from what I can provide now, I’ll take your request with me to the team. I do like to provide (at least) some level of transparency.