Clustered event bus and dynamic queues

Imported from Google Groups
1

Hi,

We’re currently trying to set up Axon to achieve the following:

  • A central application which manages the aggregates and is responsible for command handling etc. (let’s call this the core for now)

  • Several instances of another application which communicates with external systems. (let’s call this the adapters for now)

  • Communication between these two using distributed command and event buses.
    Communication using a distributed command bus is working perfectly. Basic communication using a distributed event bus (AMQP/RabbitMQ) is also working. We are now trying to solve the following problem:

  • An external system tries to communicate with our system.

  • The external system hits the load balancer and one of the instances of the adapters receives this request.

  • This adapter registers the IP address of the external system and handles the request or sends a command to the core to handle this request.

  • Something happens in the core which requires communication with an external system.

  • The core publishes an event on the distributed event bus.

  • Only the adapter which already knows the external system and its IP address should receive this event and handle it.
    Our problem right now is that last bullet. Our idea is using the following RabbitMQ topology:

  • A (direct) exchange specifically for the messages from core to the adapters.

  • External system identifiers as routing keys.

  • One queue per external system which receives the events using the aforementioned routing keys.
    Because we do not know beforehand which external system will contact us we need to dynamically create the above topology. This would give us completely separated instances of adapters which can easily be scaled out. If the load balancer decides to send messages from a specific external system to different adapters this wouldn’t be an issue either as it’ll be “first come, first serve” as to who will handle this event (which is fine for our solution).

We’ve got a small proof-of-concept which does the above using Spring AMQP but I’d like to use as much of Axon (and Axon AMQP) to recreate this so we can use all the nice stuff like annotated event listeners (instead of all the manual stuff we would have to do if we would only use Spring AMQP). I’m at a loss though as how to recreate this using Axon components (or integrate some of the manual Spring AMQP code into Axon components).

I think I would need to define clusters per external system and I think I’ll have to write a custom ClusterSelector to achieve this but honestly I’m grasping at straws here. Does anyone have any ideas, tips, or hints which could help us find a clean solution to this problem using Axon?

Regards,

Dennis Laumen

2

Hi Dennis,

there are probably many solutions for this problem, depending on the exact details of your infrastructure. The easiest solution seems to be to create a separate cluster for the handler that sends the messages to the external systems. Each instance of this cluster should connect to a different queue. This way, each instance will receive all of the events.
To achieve this, the cluster name should be different on each instance, or you need to specify a different queue name for each one by providing an AMQPConfiguration. In spring, you could do something like queueName=“myClustername-${host.name}” to achieve this. You could make the queues auto-delete to have them removed when a host disconnects.

An alternative would be to share the IP address information between nodes. A simple database table would do, but you can also use distributed collections. In that case, the different hosts can simply compete for messages on the same queue.

Hope this helps,

Allard

3

Hi Allard,

Thanks for your response!

there are probably many solutions for this problem, depending on the exact details of your infrastructure. The easiest solution seems to be to create a separate cluster for the handler that sends the messages to the external systems. Each instance of this cluster should connect to a different queue. This way, each instance will receive all of the events.

I (think I) don't want every instance to receive all of the events (but, please, correct me if I'm wrong on this one). What I'd like to have is each instance subscribe to a queue which is dedicated to receiving messages which should be delivered to a specific external system. Each instance should then be subscribed to every queue for an external system which has once contacted it. This could lead to multiple instances being subscribed to the same queues which would make them compete for the same messages (which would be fine for us).

So, in short:

- A single direct exchange routes its messages to multiple queues, one for each external system.
- A single queue might have multiple consumers, one for each instance which 'knows' the external system the queue is for.
- A single instance is consumes multiple queues, one for each external system it knows.

All the queues should be created after an external system contacts an instance for the first time.

To achieve this, the cluster name should be different on each instance, or you need to specify a different queue name for each one by providing an AMQPConfiguration. In spring, you could do something like queueName="myClustername-${host.name}" to achieve this. You could make the queues auto-delete to have them removed when a host disconnects.

Correct me if I'm wrong, but this is a bit more of a static set-up than the one I'm describing above isn't it? The thing I'm missing in the above description is creating new queues. My gut feeling is that I should somehow create new clusters with a reference to a queue (which would either be automatically created by the Cluster or I would have to manually create with Spring AMQP) and add them to the terminal but the API doesn't seem to support this.

An alternative would be to share the IP address information between nodes. A simple database table would do, but you can also use distributed collections. In that case, the different hosts can simply compete for messages on the same queue.

This is indeed our back-up solution if the above doesn't work. The above solution would be highly preferable though because (besides other advantages) it imposes less restrictions on how to implement the system's infrastructure (we won't be the only ones using this software and adding persistence to the adapters would add another requirement for the environments in which this will be deployed, we want as few of those as possible).

Thanks again, Allard!

Regards,

Dennis Laumen