I just started working on implementing dynamic tenants and I have 2 questions. I am sorry if these have been asked before (I tried to search did not find much relevant info). Your advice is much appreciated!
Question 1:
I have been working on multi-tenancy and basically have a working solution. However I want to explore using multiple event stores for different tenants. Therefore I want to try the “extension”.
I have managed to have our tenant names prefixed with “tenant-”, and have successfully added the following bean to my “AxonConfig” class. I added “axon-multitenancy” to the pom and am able to build successfully.
@Bean
public TenantConnectPredicate tenantFilterPredicate() {
return context -> context.tenantId().startsWith("tenant-");
}
However, the I have a hard time figuring out how to remove the “axon.axonserver.contexts”. Is this done by code or by modifying some config file?
(Documentation says “Note that in this case you need to remove axon.axonserver.contexts property”)
Question 2:
What does tenantFilterPredicate() really do? How do I know if it does anything differently if I run the application with the Bean vs. without?
axon.axonserver.contexts is a property in application.properties in your client Axon Framework application, if you don’t have this property then it’s fine, it’s just that you should either have this property or tenant filter predicate.
From the question I can’t conclude whether are you using Axon Server Enterprise or not, The extension currently supports Axon Server Enterprise (or Cloud) only, and the tenant filter filters out which context (out of all contexts) you want your application to connect to. This is done dynamically, as contexts may be created or deleted in runtime. In contrast axon.axonserver.contexts statically tells the application to which contexts to connect.
The extension itself is open source and it is possible to adapt to work with non-Axon Server databases by whoever wants to do it, but that requires additional work, and each database, message broker etc, need to support multitenancy.
Hi Stefan,
I learned from my team that we are using “enterprise” version with Kubernetes. Currently we are using the free version however.
When I am working on getting the free trial license, can you provide a sample log of how the tenant filter (predicate) works in action so I can know if my experiment works (or not) when I run it.
By the way, below is the version of our “server”. I did a quick search of its axonserver.properties for “axon.axonserver.contexts” and I can’t find it.
axon.axonserver.contexts is an optional property in the Axon Framework application. If you have never set it yourself, than be assured its not there and there is nothing to remove.
In this application, tenantFilter configures the application to connect to any context which name starts with tenant-. If a new context is created during runtime that matches the filter (eg. context name is: tenant-client1), the application will automatically connect to that tenant (context).
You can use your free license to start this demo and check how everything works in action.
Feel free to reach out if you have more questions.
I downloaded the “code samples” and trying to build multitenancy project and ran into a dependency issue.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.728 s
[INFO] Finished at: 2024-03-20T11:22:12-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.vaadin:vaadin-maven-plugin:24.3.7:prepare-frontend (default) on project multitenancy: Execution default of goal com.vaadin:vaadin-maven-plugin:24.3.7:prepare-frontend failed: Unable to load the mojo 'prepare-frontend' in the plugin 'com.vaadin:vaadin-maven-plugin:24.3.7' due to an API incompatibility: org.codehaus.plexus.component.repository.exception.ComponentLookupException: com/vaadin/flow/plugin/maven/PrepareFrontendMojo has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
I did a quick Google search and it sounds like a component of the vaadin plugin was compiled using Java 17, while my system is running with 11.
I got past the error about Java Runtime above and now running into a cert-related problem.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.723 s
[INFO] Finished at: 2024-03-20T13:45:50-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.12.1:compile (default-compile) on project multitenancy: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.12.1:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.12.1 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.12.1 -> org.apache.maven.shared:maven-shared-utils:jar:3.4.2: Failed to read artifact descriptor for org.apache.maven.shared:maven-shared-utils:jar:3.4.2: Could not transfer artifact org.apache.maven.shared:maven-shared-utils:pom:3.4.2 from/to central (https://artifactory-ext.digitalaviationservices.com:443/artifactory/maven-central/): transfer failed for https://artifactory-ext.digitalaviationservices.com:443/artifactory/maven-central/org/apache/maven/shared/maven-shared-utils/3.4.2/maven-shared-utils-3.4.2.pom: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
I am able to build and spring-boot:run the program.
However I am not running it in a dockerized environment; but instead I am running it on Windows with AxonServer-2023.1.1 running using “java -jar axonserver.jar” command (with license key file copied to this root directory) in hope that I can try the demo app with the most basic settings without clustering and replication, etc. In other words, I want to run in with a standard mode, single node first. I noticed there are settings in “src/main/docker” folder, especially the “cluster-template.yml” and “docker-comose.yml” to set up 3-node. I am tempting to simplify them to drop the 3-node settings to 1-node but want to run it by you first.
Thanks
-Peter
2024-03-20 16:34:30.571 INFO 22696 --- [ main] io.axoniq.axonserver.AxonServer : Starting AxonServer using Java 17.0.10 on A6327613 with PID 22696 (C:\Users\wr911f\Downloads\AxonServer-2023.1.1\axonserver.jar started by wr911f in C:\Users\wr911f\Downloads\AxonServer-2023.1.1)
2024-03-20 16:34:30.577 INFO 22696 --- [ main] io.axoniq.axonserver.AxonServer : No active profile set, falling back to 1 default profile: "default"
2024-03-20 16:34:33.337 INFO 22696 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8024 (http)
2024-03-20 16:34:33.464 INFO 22696 --- [ main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2024-03-20 16:34:34.119 WARN 22696 --- [ main] o.f.core.internal.command.DbMigrate : outOfOrder mode is active. Migration of schema "PUBLIC" may not be reproducible.
2024-03-20 16:34:35.848 INFO 22696 --- [ main] A.i.a.a.licensing.LicenseManager : License loaded and validated for 'AxonServer'.
2024-03-20 16:34:35.848 INFO 22696 --- [ main] io.axoniq.axonserver.AxonServer : Axon Server version 2023.1.1
2024-03-20 16:34:39.588 INFO 22696 --- [ main] i.a.a.e.c.i.MessagingClusterServer : Axon Server Cluster Server started on port: 8224 - no SSL
2024-03-20 16:34:39.715 INFO 22696 --- [ main] io.axoniq.axonserver.grpc.Gateway : Axon Server Gateway started on port: 8124 - no SSL
2024-03-20 16:34:39.715 INFO 22696 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8024 (http) with context path ''
2024-03-20 16:34:39.744 INFO 22696 --- [ main] io.axoniq.axonserver.AxonServer : Started AxonServer in 9.736 seconds (JVM running for 10.247)
2024-03-20 16:40:12.448 INFO 22696 --- [nio-8024-exec-9] i.a.a.a.c.r.ContextAdminRequestProcessor : [<anonymous>] Request to list contexts.
2024-03-20 16:41:02.929 INFO 22696 --- [nio-8024-exec-9] i.a.a.a.c.r.ContextAdminRequestProcessor : [<anonymous>] Request to list contexts.
And on another terminal I got error from the app saying failed to connect
2024-03-20T16:44:21.257-07:00 INFO 13392 --- [multitenant-example] [ main] org.atmosphere.cpr.AtmosphereFramework : Atmosphere Framework 3.0.4.slf4jvaadin1 started.
2024-03-20T16:44:21.257-07:00 INFO 13392 --- [multitenant-example] [ main] org.atmosphere.cpr.AtmosphereFramework : Installed AtmosphereInterceptor Track Message Size Interceptor using | with priority BEFORE_DEFAULT
2024-03-20T16:44:21.274-07:00 INFO 13392 --- [multitenant-example] [ main] c.v.f.s.DefaultDeploymentConfiguration : Vaadin is running in production mode.
2024-03-20T16:44:21.323-07:00 INFO 13392 --- [multitenant-example] [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port 8080 (http) with context path ''
2024-03-20T16:44:21.557-07:00 INFO 13392 --- [multitenant-example] [ main] i.a.a.c.impl.AxonServerManagedChannel : Requesting connection details from localhost:8124
2024-03-20T16:44:21.825-07:00 WARN 13392 --- [multitenant-example] [ main] i.a.a.c.impl.AxonServerManagedChannel : Connecting to AxonServer node [localhost:8124] failed.
io.grpc.StatusRuntimeException: NOT_FOUND: [AXONIQ-1302] _admin: not found in any replication group
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.axoniq.axonserver.grpc.control.PlatformServiceGrpc$PlatformServiceBlockingStub.getPlatformServer(PlatformServiceGrpc.java:250) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.connectChannel(AxonServerManagedChannel.java:115) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.createConnection(AxonServerManagedChannel.java:335) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.ensureConnected(AxonServerManagedChannel.java:300) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel.getState(AxonServerManagedChannel.java:227) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.ContextConnection.ensureConnected(ContextConnection.java:237) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.impl.ContextConnection.connect(ContextConnection.java:165) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.AxonServerConnectionFactory.connect(AxonServerConnectionFactory.java:166) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at org.axonframework.axonserver.connector.AxonServerConnectionManager.createConnection(AxonServerConnectionManager.java:129) ~[axon-server-connector-4.9.3.jar:4.9.3]
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[na:na]
at org.axonframework.axonserver.connector.AxonServerConnectionManager.getConnection(AxonServerConnectionManager.java:125) ~[axon-server-connector-4.9.3.jar:4.9.3]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.getTenantsAPI(AxonServerTenantProvider.java:160) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.getInitialTenants(AxonServerTenantProvider.java:104) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.start(AxonServerTenantProvider.java:88) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.lifecycle.Lifecycle$LifecycleRegistry.lambda$onStart$0(Lifecycle.java:68) ~[axon-messaging-4.9.3.jar:4.9.3]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[na:na]
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
at java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:662) ~[na:na]
at org.axonframework.config.DefaultConfigurer.invokeLifecycleHandlers(DefaultConfigurer.java:1058) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.config.DefaultConfigurer.invokeStartHandlers(DefaultConfigurer.java:1004) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.config.DefaultConfigurer$ConfigurationImpl.start(DefaultConfigurer.java:1156) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.spring.config.SpringAxonConfiguration.start(SpringAxonConfiguration.java:76) ~[axon-spring-4.9.3.jar:4.9.3]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:288) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:471) ~[spring-context-6.1.4.jar:6.1.4]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:260) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:205) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:978) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:627) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-3.2.3.jar:3.2.3]
at io.axoniq.multitenancy.MultitenancyExampleApplication.main(MultitenancyExampleApplication.java:12) ~[classes/:na]
2024-03-20T16:44:21.825-07:00 INFO 13392 --- [multitenant-example] [ main] i.a.a.c.impl.AxonServerManagedChannel : Failed to get connection to AxonServer. Scheduling a reconnect in 2000ms
2024-03-20T16:44:21.825-07:00 ERROR 13392 --- [multitenant-example] [ main] o.a.e.m.a.AxonServerTenantProvider : Error while getting initial tenants
java.util.concurrent.CompletionException: io.grpc.StatusRuntimeException: UNAVAILABLE
at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118) ~[na:na]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.getTenantsAPI(AxonServerTenantProvider.java:163) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.getInitialTenants(AxonServerTenantProvider.java:104) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.start(AxonServerTenantProvider.java:88) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
at org.axonframework.lifecycle.Lifecycle$LifecycleRegistry.lambda$onStart$0(Lifecycle.java:68) ~[axon-messaging-4.9.3.jar:4.9.3]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[na:na]
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na]
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
at java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:662) ~[na:na]
at org.axonframework.config.DefaultConfigurer.invokeLifecycleHandlers(DefaultConfigurer.java:1058) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.config.DefaultConfigurer.invokeStartHandlers(DefaultConfigurer.java:1004) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.config.DefaultConfigurer$ConfigurationImpl.start(DefaultConfigurer.java:1156) ~[axon-configuration-4.9.3.jar:4.9.3]
at org.axonframework.spring.config.SpringAxonConfiguration.start(SpringAxonConfiguration.java:76) ~[axon-spring-4.9.3.jar:4.9.3]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:288) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:471) ~[spring-context-6.1.4.jar:6.1.4]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:260) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:205) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:978) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:627) ~[spring-context-6.1.4.jar:6.1.4]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354) ~[spring-boot-3.2.3.jar:3.2.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-3.2.3.jar:3.2.3]
at io.axoniq.multitenancy.MultitenancyExampleApplication.main(MultitenancyExampleApplication.java:12) ~[classes/:na]
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE
at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel$FailingCall.start(AxonServerManagedChannel.java:413) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:335) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:311) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:299) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.grpc.stub.ClientCalls.asyncServerStreamingCall(ClientCalls.java:91) ~[grpc-stub-1.59.1.jar:1.59.1]
at io.axoniq.axonserver.grpc.admin.ContextAdminServiceGrpc$ContextAdminServiceStub.getContexts(ContextAdminServiceGrpc.java:402) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at io.axoniq.axonserver.connector.admin.impl.AdminChannelImpl.getAllContexts(AdminChannelImpl.java:365) ~[axonserver-connector-java-2023.2.0.jar:2023.2.0]
at org.axonframework.extensions.multitenancy.autoconfig.AxonServerTenantProvider.getTenantsAPI(AxonServerTenantProvider.java:162) ~[axon-multitenancy-spring-boot-autoconfigure-4.9.2.jar:4.9.2]
... 28 common frames omitted
I am able to connect and run the demo GUI app. The reason I got into the error above is due to the fact that Axon Server was not fully configured. I found this when I went to “locahost:8024” and noticed the web page was asking “It appears that you have a fresh installation. How would you like to initialize this node?”
I then chose “Start as standalone node or first node”, and Without context name “default”. After that I hit “Finish” and I was able to connect and ran the sample app.
That’s the good part.
Now the bad part is, I don’t know how to reset and clear all the tenants I created. I don’t think this version of Axon Server works with “devmode” reset. In addition I tried using cli “purge-events” and that does not work either.
And since the axon bus is full of old events, when I spring-boot the sample app again/subsequently, it fails with SQL errors. I guess if I can reset Axon and clear data from the previous run, then I am good (for now) to try and understand the sample app.
Thanks for the offer Stefan. I am able to reset Axon and able to run the demo app (again) now. I will evaluate it and touch base with my team. I will circle back.