Axon application scalability on kubernetes pods

Axon Framework
,
1

I am deploying Axon spring boot application on kubernetes cluster(AWS EKS). Two pods are launched and want to scale appliaction. I have added one configuration in sprinboot application -

axon.distributed.enabled=true

and added jgroup-extenstion dependency in class path. after deployment I can see jgroup cluster created.

Pod 1 -

GMS: address=abc-64b857d647-w7nbt-33031, cluster=Axon, physical address=10.134.1.216:7800
-------------------------------------------------------------------
17:47:07.382 INFO  org.jgroups.protocols.pbcast.GMS -
                    abc-64b857d647-w7nbt-33031: no members discovered after 3002 ms: creating cluster as coordinator
17:47:07.391 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Local segment (abc-64b857d647-w7nbt-33031) joined the cluster. Broadcasting configuration.
17:47:07.391 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Broadcasting membership from nettingeq-64b857d647-w7nbt-33031
17:47:07.391 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Sending my configuration to all nodes.
17:47:07.586 INFO  com.abc.Application -
                    Started Application in 36.02 seconds (JVM running for 41.156)

Pod 2 -

GMS: address=abc-64b857d647-xb4hj-27405, cluster=Axon, physical address=10.134.2.66:7800
-------------------------------------------------------------------
17:46:31.436 INFO  org.jgroups.protocols.pbcast.GMS -
                    abc-64b857d647-xb4hj-27405: no members discovered after 3002 ms: creating cluster as coordinator
17:46:31.445 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Local segment (abc-64b857d647-xb4hj-27405) joined the cluster. Broadcasting configuration.
17:46:31.445 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Broadcasting membership from nettingeq-64b857d647-xb4hj-27405
17:46:31.445 INFO  org.axonframework.extensions.jgroups.commandhandling.JGroupsConnector -
                    Sending my configuration to all nodes.
17:46:31.632 INFO  com.abc.Application -
                    Started Application in 32.789 seconds (JVM running for 37.135)

Once send any command one node doesn’t know the command is handle by other node.
is there any configuration I need to add so that they can join the jgroup cluster and do gossip?

2

Can you share the whole configuration? As you can see in the reference guide, there might be some more properties that need to be set.

3

Hi @Gerard
I used only this configuration axon.distributed.enabled=true . Rest I have not used and keep default

axon.distributed.jgroups.bind-addr=GLOBAL
# the port to bind the local instance to
axon.distributed.jgroups.bind-port=7800

# the name of the JGroups Cluster to connect to
axon.distributed.jgroups.cluster-name=Axon

# the JGroups Configuration file to configure JGroups with
axon.distributed.jgroups.configuration-file=default_tcp_gossip.xml

# The IP and port of the Gossip Servers (comma separated) to connect to
axon.distributed.jgroups.gossip.hosts=localhost[12001]
# when true, will start an embedded Gossip Server on bound to the port of the first mentioned gossip host.
axon.distributed.jgroups.gossip.auto-start=false

I am not sure about this config axon.distributed.jgroups.gossip.hosts=localhost[12001] is this configuration to connect to servers and join in group?

As I am deplyoing the application on AWS eks cluster, kuberenetes pods creates randomly with different address which is dynamic. so I am not sure I would be able to give the addresses of pods in configuration. Any suggestion or refrence will be highly appreciated How I can scale it on kubernetes? Currently What I am seeing, when deploying application, each pod assigning themself as cordinator/Leader instead of joining as a group.

4

I never used Jgroups, there is a demo project which is a few years old but might be helpful. This article might help with Jgroups specifically.

There should be some way multiple instances can connect to each other. In the config file of the demo project are addresses of both hosts.

5

Thanks @Gerard for the resources. I will check them. there is one more option for distributing command bus with spring cloud extension. Is there any recommendation when to use Jgroup or Spring cloud for distributing command bus. Currently I have a springboot application with Axon. Can I get Refrence of the springcloud extenstion to scale it on k8 instances.

6

It depends a lot on what you already have running I think. For Spring Cloud you need Eureka or Consul, or something similar.

By far the easiest way for distributed commands is Axon Server through. As gRPC is bidirectional, it’s enough for the clients to connect to Axon Server. For the other solutions the instances have to be able to find and connect to each other by other means.