Microservice implementing the axon framework shuts down automatically as in memory H2 database does not get the axon related tables created. So due to this issue the microservice is not started off at all

Hi,
I have a microservice which is using axon framework and having in memory database H2 configurations in it’s properties file. When I trying to deploy this microservice (in my AWS EKS where axon server is also running in same namespace), then during starting up it fails. It is able to connect to it’s axon server while starting up but then suddenly it goes down as it is not able to find the table “TOKEN_ENTRY” in the in-memory H2 database of itself. So the microservice is not started off. Please help me on this. Please find below the logs too…

f.TestSpringbootStarterApplication : Starting TestSpringbootStarterApplication v0.1 using Java 11.0.7 on test-app-554bb59646-gv2kf with PID 1

f.TestSpringbootStarterApplication : No active profile set, falling back to default profiles: default

.s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.

.s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 396 ms. Found 1 JPA repository interfaces.

o.s.c.a.ConfigurationClassPostProcessor : Cannot enhance @Configuration bean definition ‘org.axonframework.springboot.autoconfig.EventProcessingAutoConfiguration’ since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as ‘static’.

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#4a83a74a#33’ of type [org.axonframework.spring.config.annotation.SpringContextParameterResolverFactoryBuilder$ClasspathParameterResolverFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#4a83a74a#33’ of type [org.axonframework.messaging.annotation.MultiParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#5032714f#33’ of type [org.axonframework.spring.config.annotation.SpringBeanDependencyResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#1b84f475’ of type [org.axonframework.spring.config.annotation.SpringBeanParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-parameter-resolver-factory’ of type [org.axonframework.spring.config.ApplicationContextLookupParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-parameter-resolver-factory’ of type [org.axonframework.messaging.annotation.MultiParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#7dda48d9’ of type [org.axonframework.spring.config.annotation.SpringHandlerDefinitionBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#7dda48d9’ of type [org.axonframework.messaging.annotation.MultiHandlerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#16ce702d’ of type [org.axonframework.spring.config.annotation.SpringHandlerEnhancerDefinitionBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#16ce702d’ of type [org.axonframework.messaging.annotation.MultiHandlerEnhancerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-handler-definition’ of type [org.axonframework.messaging.annotation.MultiHandlerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)

o.apache.catalina.core.StandardService : Starting service [Tomcat]

org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.53]

o.a.c.c.C.[.[.[/jiraagentservice] : Initializing Spring embedded WebApplicationContext

w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 9901 ms

com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting…

com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.

o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]

org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.32.Final

o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}

org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect

o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]

j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’

o.a.serialization.ChainingConverter : ContentTypeConverter of type [class org.axonframework.serialization.xml.XomToStringConverter] is ignored. It seems to rely on a class that is not available in the class loader: nu/xom/Document

o.a.serialization.ChainingConverter : ContentTypeConverter of type [class org.axonframework.serialization.xml.InputStreamToXomConverter] is ignored. It seems to rely on a class that is not available in the class loader: nu/xom/ParsingException

JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning

i.a.a.c.impl.AxonServerManagedChannel : Requesting connection details from 10.149.50.62:8124

i.a.a.c.impl.AxonServerManagedChannel : Successfully connected to 10.149.50.62:8124

i.a.a.connector.impl.ControlChannelImpl : Connected instruction stream for context ‘default’. Sending client identification

i.a.a.c.command.impl.CommandChannelImpl : CommandChannel for context ‘default’ connected, 0 command handlers registered

i.a.a.c.command.impl.CommandChannelImpl : Registered handler for command ‘com.test…commonservice.commands.CancelOrderCommand’ in context ‘default’

i.a.a.c.command.impl.CommandChannelImpl : Registered handler for command ‘com.test…OrderService.command.api.command.CreateOrderCommand’ in context ‘default’

i.a.a.c.command.impl.CommandChannelImpl : Registered handler for command ‘com.test…commonservice.commands.CompleteOrderCommand’ in context ‘default’

Security framework of XStream not explicitly initialized, using predefined black list on your own risk.

[d.api.events]-0] o.a.e.TrackingEventProcessor : Worker assigned to segment Segment[0/0] for processing

[agaProcessor]-0] o.a.e.TrackingEventProcessor : Worker assigned to segment Segment[0/0] for processing

[d.api.events]-0] o.a.e.TrackingEventProcessor : Using current Thread for last segment worker: TrackingSegmentWorker{processor=com.test…OrderService.command.api.events, segment=Segment[0/0]}

[agaProcessor]-0] o.a.e.TrackingEventProcessor : Using current Thread for last segment worker: TrackingSegmentWorker{processor=OrderProcessingSagaProcessor, segment=Segment[0/0]}

[d.api.events]-0] o.a.e.TrackingEventProcessor : Fetched token: null for segment: Segment[0/0]

[agaProcessor]-0] o.a.e.TrackingEventProcessor : Fetched token: null for segment: Segment[0/0]

[ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ‘/jiraagentservice’

[ main] f.TestSpringbootStarterApplication : Started TestSpringbootStarterApplication in 30.503 seconds (JVM running for 32.801)

[nio-8080-exec-2] o.a.c.c.C.[.[.[/jiraagentservice] : Initializing Spring DispatcherServlet ‘dispatcherServlet’

[nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Initializing Servlet ‘dispatcherServlet’

[nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Completed initialization in 1 ms

[ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete

[tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown : Graceful shutdown complete

[ionShutdownHook] o.a.e.TrackingEventProcessor : Shutdown state set for Processor ‘OrderProcessingSagaProcessor’.

[ionShutdownHook] o.a.e.TrackingEventProcessor : Shutdown state set for Processor ‘com.test.OrderService.command.api.events’.

[ Thread-1] o.a.e.TrackingEventProcessor : Processor ‘OrderProcessingSagaProcessor’ awaiting termination…

[ Thread-2] o.a.e.TrackingEventProcessor : Processor ‘com.test…OrderService.command.api.events’ awaiting termination…

[agaProcessor]-0] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 42102, SQLState: 42S02

2024-04-16 13:54:55.452 ERROR 1 — [agaProcessor]-0] o.h.engine.jdbc.spi.SqlExceptionHelper : Table “TOKEN_ENTRY” not found; SQL statement:

update token_entry set timestamp=? where processor_name=? and segment=? and owner=? [42102-200]

2024-04-16 13:54:55.461 WARN 1 — [agaProcessor]-0] o.a.e.TrackingEventProcessor : Error occurred. Starting retry mode.

javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not prepare statement

at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1692) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

at org.axonframework.eventhandling.tokenstore.jpa.JpaTokenStore.extendClaim(JpaTokenStore.java:211) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.eventhandling.TrackingEventProcessor.lambda$processBatch$12(TrackingEventProcessor.java:429) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.common.transaction.TransactionManager.executeInTransaction(TransactionManager.java:47) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:428) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:294) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1005) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1149) ~[axon-messaging-4.5.3.jar!/:4.5.3]

at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

Caused by: org.hibernate.exception.SQLGrammarException: could not prepare statement

at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) ~[hibernate-core-5.4.32.Final.jar!/:5.4.32.Final]

I am using postgresql so here is the script to create token_entry:

CREATE TABLE token_entry
(
  processor_name character varying(255) NOT NULL,
  segment integer NOT NULL,
  owner character varying(255),
  "timestamp" character varying(255) NOT NULL,
  token bytea,
  token_type character varying(255),
  CONSTRAINT token_entry_pkey PRIMARY KEY (processor_name, segment)
)

Do you need a liveliness probe for this container to keep it up? It looks like everything comes up and then there is a log about initiating a graceful shutdown.

Hi Ben,
sorry for late reply.

Yes there is liveness probe for this container. Yes everything came up and then it went down suddenly. After adding springboot actuator jar we got rid of it. Now all is working fine.

Thank you so much Ben for checking out on this issue and helping me. :slight_smile: