Load balanced websocket service

Axon Framework
#1

Hi there,

We recently switched from Axon 2 to Axon 4. We have our application split in multiple services. One of these services have websocket connections with clients. Each client is an aggregate within Axon.

Now our issue is that we have this specific service scaled. Each instance has it’s own set of websocket sessions. Is there any way we can tell Axon to send events to specific instances/segments? Or is it maybe possible to send the events to all segments instead, we already have our own logic to deal with aggregates that don’t have a websocket connection in the instance.

#2

Hi Paul,

Axon enabled applications are usually communicating with each other by sharing messages (commands, events, queries). These messages are representing the core API of these services. The messages are shared via distributed buses/gateways: CommandBus/CommandGateway, EventBus/EventGateway, QueryBus/QueryGateway. Axon Server, being a message broker, implements all three types of busses, and it will route messages out of the box. By default, it utilizes a Consistent Hashing algorithm making sure that your commands (belonging to the same aggregate) will be handled by the same instance of your application (for example, command handler in aggregate)

As I understand you have a WebSocket adapter/component surrounding your core API and exposing WebSocket endpoints. Personally, this WebSocket API is more convenient as a user-friendly API that is exposed to your customers and it does not have to be used for inter-service communication (you can use core API for this).

1
11420×1100 58.7 KB

This is a related topic that you might find iteresting:

Load balancing websocket services is not an Axon specific topic. You should be able to use service discovery and registration services (like Eureka for example) to achieve this.

Best,
Ivan

#3

I made a diagram to visualize our architecture

Distributed Architecture
Distributed Architecture419×652 18.2 KB

As can be seen the backend consists out of a couple applications:

  • Domain: This application runs our aggregates.
  • WebSocket API: This application translates messages from our WebSocket API to Axon events, also listens for events that should be sent to specific clients.
  • Some other services: We have some other services that communicate using Axon.

The loadbalancing is done by an external cloud service. In the diagram we have two instances of the WebSocket API. The clients are balanced over these two instances (can be scaled when needed).

The issue we are having is that we have to split the segment in two for these instances (or more when we scale). Because of the segmenting we can’t tell that events of an aggregate should specifically be received by one of the two instances. Ideally we should be able to tell Axon to just send the events to both instances and let us handle the rest.

1 Like
#4

Nice, thanks for sharing! Looks nice!

WebSocketAPI application is essentially an adapter that maps user input/message to axon command (which will be handled by an Aggregate) and axon event (or subscribing query if possible) to a WebSocket message back to the client application; Command = input; Query = output;

This application (WebSocketAPI) is a WebSocket Server that exposes endpoints. If you have multiple instances of your WebSocket Server, then every instance needs to know the sessions that exist on other instances.

Therefore you need to use a broker relay (not the in-memory broker given by spring).

There is a chance that I still did not understand your question :(, but it looks like this is not related to Axon. Axon Server will route commands, events, and queries in a good manner acting as a good broker for these three types of messages.

For web socket messages you would need some production-ready broker most probably. Via a protocol like STOMP (Simple Text Oriented Message Protocol) or AMQP (Advanced Message Queuing Protocol).

#5

This diagram reflects a project I’ve worked on in the past actually. Not so much an answer to your immediate problem @paulhobbel, but Axon has introduced subscription queries in Axon 3 to keep your instances up to date automatically. Axon Server automatically deals with those in a distributed environment as well, ensuring you can leave all the routing in Axon’s hands.

I understand you are comfortable doing it yourself. If so, by all means. Just wanted to point out that I assume subscription queries would be the up-to-date solution nowadays.

That’s my 2 cents.