Multitenancy Extension - shutdown handler

Axon Framework
1

I use the multitenancy extension in version 4.8 and trying to switch to 4.9.3. I adjusted things and the application starts and working fine but I see many errors saying that “token_entry does not exist” when I kill it (e.g. in Intellij). I noticed that it is caused by the shutdown handler in AxonServerTenentProvider class (in the 4.8 version this handler didn’t exist).

It looks like it does not consider the tenant and its data source so it probably chose the default data source (schema: public). I use Postgres Hikari data source with different schemas per tenant (changing schema in URL in DataSourceResolver). Below is one of these errors:


2024-07-31 14:46:06.640 ERROR [,] 7888 --- [ions@default]-0] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: relation "token_entry" does not exist
 Pozycja: 8
2024-07-31 14:46:06.652 INFO [,] 7888 --- [ions@default]-0] o.a.eventhandling.pooled.Coordinator : An exception occurred during the abort of work package [0] on [projections@default] processor.

java.util.concurrent.CompletionException: jakarta.persistence.PersistenceException: Converting `org.hibernate.exception.SQLGrammarException` to JPA `PersistenceException` : JDBC exception executing SQL [update token_entry set owner=null where owner=? and processor_name=? and segment=?]
 at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
 at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:823)
 at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
 at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.abortWorkPackage(Coordinator.java:1060)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.lambda$abortWorkPackages$9(Coordinator.java:839)
 at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
 at java.base/java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3612)
 at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
 at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
 at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
 at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:662)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.abortWorkPackages(Coordinator.java:840)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.run(Coordinator.java:709)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.lambda$scheduleCoordinationTask$20(Coordinator.java:1014)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: jakarta.persistence.PersistenceException: Converting `org.hibernate.exception.SQLGrammarException` to JPA `PersistenceException` : JDBC exception executing SQL [update token_entry set owner=null where owner=? and processor_name=? and segment=?]
 at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:165)
 at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:175)
 at org.hibernate.query.sqm.internal.QuerySqmImpl.executeUpdate(QuerySqmImpl.java:709)
 at org.axonframework.eventhandling.tokenstore.jpa.JpaTokenStore.releaseClaim(JpaTokenStore.java:172)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.lambda$null$24(Coordinator.java:1062)
 at org.axonframework.common.transaction.TransactionManager.executeInTransaction(TransactionManager.java:47)
 at org.axonframework.extensions.multitenancy.TenantWrappedTransactionManager.executeInTransaction(TenantWrappedTransactionManager.java:73)
 at org.axonframework.eventhandling.pooled.Coordinator$CoordinationTask.lambda$abortWorkPackage$25(Coordinator.java:1060)
 at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
 ... 21 common frames omitted
Caused by: org.hibernate.exception.SQLGrammarException: JDBC exception executing SQL [update token_entry set owner=null where owner=? and processor_name=? and segment=?]
 at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:89)
 at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:56)
 at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109)
 at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:95)
 at org.hibernate.sql.exec.internal.StandardJdbcMutationExecutor.execute(StandardJdbcMutationExecutor.java:97)
 at org.hibernate.query.sqm.internal.SimpleUpdateQueryPlan.executeUpdate(SimpleUpdateQueryPlan.java:93)
 at org.hibernate.query.sqm.internal.QuerySqmImpl.doExecuteUpdate(QuerySqmImpl.java:728)
 at org.hibernate.query.sqm.internal.QuerySqmImpl.executeUpdate(QuerySqmImpl.java:698)
 ... 27 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: relation "token_entry" does not exist
 Pozycja: 8
 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
 at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
 at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:152)
 at org.hibernate.sql.exec.internal.StandardJdbcMutationExecutor.execute(StandardJdbcMutationExecutor.java:84)
 ... 30 common frames omitted
2

Hi, is this the demo app that you have been running, or your own implementation?

In the demo app this happens, because demo uses in memory token store, for real usages you need persistent token store.
Even better in our latest 4.10.1 version we support persistant streams, Axon Server takes care of everything you don’t need token store, and number of tenants are more scalable

3

Not demo app.It is real app (as I mentioned I am using postgres db).

In case of version 4.10.x there is a mess in maven repo e.g. there is no bom artifact with that version (https://mvnrepository.com/artifact/org.axonframework/axon-bom) so I don’t want to use it now.

Anyway Currently I am using this mentioned 4.9.3 with custom AxonServerTenantProvider where removed onShutdown from registerLifecycleHandlers

            object : AxonServerTenantProvider(contexts, tenantConnectPredicate, axonServerConnectionManager) {

                override fun registerLifecycleHandlers(@Nonnull lifecycle: LifecycleRegistry) {
                    lifecycle.onStart(Phase.INSTRUCTION_COMPONENTS + 10, Runnable { start() })
//                    TODO: waiting for Axon feedback
//                    onShutdown causing issues (cannot found token_entry table)
//                    lifecycle.onShutdown(Phase.INSTRUCTION_COMPONENTS + 10, Runnable { shutdown() })
                }
            }
4

I’m struggling to understand the issue here.

You see this errors only during the shutdown of the whole app?
Do they occure again when you restart app?

5

It happens only on shutdown - all the time. Actually don’t now what is the purpose of this onshutdown because even without it token table is properly updated means owner column is cleaned up

6

All Axon Framework infrastructure components have lifecycle management methods. The AxonServerTenentProvider#shutdown is one of those.
These shutdown hooks are registered in order within Axon Framework’s configuration.

In doing so, we ensure the infrastructure components are shutdown in the right order.
This is, in turn, intended to ensure the users application (so, your application) shuts down nicely. So, without partial message handling or unhelpful exceptions.

In the stack trace you’ve shared, it might become apparent that the shutdown process is stopping the PooledStreamingEventProcessor(s) you have in your application

The “unhelpful exception”-part might be the problem at hand, though.

However, I am on the same page as @stefand here. If you’re using PosgreSQL as your database, you should have a JpaTokenStore or JdbcTokenStore backing your Event Processors. After checking the stack trace again, I spot the JpaTokenStore actually. Furthermore, you would definitely have the token_entry table, while the exception clearly states this is not the case.

So, perhaps you can elaborate on how you’ve configured, if it all, the token_entry table, @KaeF?

7

@Steven_van_Beelen
token_entry table is properly created by flyway (as any other table) for each tenant (each tenant has separate schema in the same db) when the application starts and it is working properly (no special treatment or configuration for this table). The only issue is this shutdown handler where it just does not consider tenant data source I think (in previous versions of the multitenancy extension there wasn’t this handler so there was no issue). I didn’t investigate it deeper but it did not go through tenantDataSourceResolver so it probably takes the default data source without schema defined so probably try to find this table in the public schema where this table doesn’t exist.

1 Like
8

Thanks for that, @KaeF.
That sounds like a very reasonable assumption to me.

During the shutdown hook, there is no active tenant at all.
Hence, the default source is picked, which doesn’t contain the token_entry table.

I’ll discuss this with @stefand in due time and we’ll come up with a solution.

In the meantime, your workaround would do the trick.
Although there is a small window of opportunity of partially handled messages and unhelpful exceptions as shared in my previous post, the chances that these break your system are extremely minimal.

Thanks again, @KaeF, and thanks for your patients :pray:

9

@KaeF I was not able to reproduce the issue at the latest v4.10.1 version.
What you are experiencing may be wrong order of shutdown. Can you try with latest version and see if issue persists?

10

Yes same in v4.10.1

shutdown_issue