My springboot microservices using axon framework goes down after starting off successfully (in AWS EKS cluster) without any exception being thrown

deepraj · April 18, 2024, 1:21pm

Hi,
I have a microservice which is using axon framework and having in memory database H2 configurations in it’s properties file. When I trying to deploy this microservice (in my AWS EKS where axon server is also running in same namespace), it starts off successfully, but then suddenly it goes down and shuts itself. Though it is successfully able to connect to the axon server (during start up) which is running in the same namespace as microservice, this connection to axon server I have verified it in logs too. Please find below logs too, please help me.

2024-04-18 09:15:15.328 INFO 1 — [ 2024-04-18 09:15:15.333 INFO 1 — [ 2024-04-18 09:15:21.532 INFO 1 — [ 2024-04-18 09:15:21.931 INFO 1 — [ 2024-04-18 09:15:22.337 INFO 1 — [ 2024-04-18 09:15:23.244 INFO 1 — [ 2024-04-18 09:15:23.244 INFO 1 — [ 2024-04-18 09:15:23.324 INFO 1 — [ 2024-04-18 09:15:23.324 INFO 1 — [ 2024-04-18 09:15:23.440 INFO 1 — [ 2024-04-18 09:15:23.440 INFO 1 — [ 2024-04-18 09:15:23.533 INFO 1 — [ 2024-04-18 09:15:23.534 INFO 1 — [ 2024-04-18 09:15:23.535 INFO 1 — [ 2024-04-18 09:15:23.536 INFO 1 — [ 2024-04-18 09:15:23.538 INFO 1 — [ 2024-04-18 09:15:24.828 INFO 1 — [ 2024-04-18 09:15:24.840 INFO 1 — [ 2024-04-18 09:15:24.841 INFO 1 — [ 2024-04-18 09:15:25.145 INFO 1 — [ 2024-04-18 09:15:25.145 INFO 1 — [ 2024-04-18 09:15:25.346 INFO 1 — [ 2024-04-18 09:15:26.441 INFO 1 — [ 2024-04-18 09:15:26.449 INFO 1 — [ 2024-04-18 09:15:28.327 INFO 1 — [ 2024-04-18 09:15:28.535 INFO 1 — [ 2024-04-18 09:15:29.227 INFO 1 — [ 2024-04-18 09:15:29.829 INFO 1 — [ 2024-04-18 09:15:32.842 INFO 1 — [ 2024-04-18 09:15:32.928 INFO 1 — [ 2024-04-18 09:15:34.539 WARN 1 — [ 2024-04-18 09:15:36.761 INFO 1 — [ 2024-04-18 09:15:36.762 INFO 1 — [ 2024-04-18 09:15:40.027 INFO 1 — [ 2024-04-18 09:15:41.026 INFO 1 — [ 2024-04-18 09:15:41.040 INFO 1 — [ 2024-04-18 09:15:41.052 INFO 1 — [ 2024-04-18 09:15:41.124 INFO 1 — [ Security framework of XStream 2024-04-18 09:15:42.029 INFO 1 — [ 2024-04-18 09:15:42.132 INFO 1 — [ 2024-04-18 09:15:42.539 INFO 2024-04-18 09:15:42.542 INFO 2024-04-18 09:15:42.546 INFO 2024-04-18 09:16:13.743 INFO 2024-04-18 09:16:13.743 INFO 2024-04-18 09:16:13.745 INFO 2024-04-18 09:16:53.560 INFO 2024-04-18 09:16:53.568 INFO 2024-04-18 09:16:53.626 INFO 2024-04-18 09:16:53.627 INFO 1 — [ 2024-04-18 09:16:53.870 INFO 2024-04-18 09:16:53.871 INFO 2024-04-18 09:16:54.029 INFO 2024-04-18 09:16:54.518 INFO 2024-04-18 09:16:54.521 INFO 2024-04-18 09:16:54.524 INFO main] f.MyPOCSpringbootStarterApplication : Starting MyPOCSpringbootStarterApplication v1.0-SNAPSHOT using Java 11.0.7 on jenkinsagentservice-jenkinsagentservice-8ddf4b55d-nh4xv with PID 1 (/home/moddel/jenkinsagentservice-exec.jar started by moddel in /home/moddel)
main] f.MyPOCSpringbootStarterApplication : No active profile set, falling back to default profiles: default
main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 390 ms. Found 1 JPA repository interfaces.
main] o.s.c.a.ConfigurationClassPostProcessor : Cannot enhance @Configuration bean definition ‘org.axonframework.springboot.autoconfig.EventProcessingAutoConfiguration’ since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as ‘static’.
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#5afa3c9#33’ of type [org.axonframework.spring.config.annotation.SpringContextParameterResolverFactoryBuilder$ClasspathParameterResolverFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#5afa3c9#33’ of type [org.axonframework.messaging.annotation.MultiParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#7ce026d3#33’ of type [org.axonframework.spring.config.annotation.SpringBeanDependencyResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#1608bcbd’ of type [org.axonframework.spring.config.annotation.SpringBeanParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-parameter-resolver-factory’ of type [org.axonframework.spring.config.ApplicationContextLookupParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-parameter-resolver-factory’ of type [org.axonframework.messaging.annotation.MultiParameterResolverFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#344f4dea’ of type [org.axonframework.spring.config.annotation.SpringHandlerDefinitionBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#344f4dea’ of type [org.axonframework.messaging.annotation.MultiHandlerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#47f9738’ of type [org.axonframework.spring.config.annotation.SpringHandlerEnhancerDefinitionBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘(inner bean)#47f9738’ of type [org.axonframework.messaging.annotation.MultiHandlerEnhancerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] trationDelegate$BeanPostProcessorChecker : Bean ‘__axon-handler-definition’ of type [org.axonframework.messaging.annotation.MultiHandlerDefinition] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.53]
main] o.a.c.c.C.[.[.[/jenkinsagentservice] : Initializing Spring embedded WebApplicationContext
main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 9516 ms
main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting…
main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
main] o.s.b.a.h2.H2ConsoleAutoConfiguration : H2 console available at ‘/h2-console’. Database available at ‘jdbc:h2:file:~/data/shipmentDB’
main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.32.Final
main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit ‘default’
main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
main] o.a.serialization.ChainingConverter : ContentTypeConverter of type [class org.axonframework.serialization.xml.XomToStringConverter] is ignored. It seems to rely on a class that is not available in the class loader: nu/xom/Document
main] o.a.serialization.ChainingConverter : ContentTypeConverter of type [class org.axonframework.serialization.xml.InputStreamToXomConverter] is ignored. It seems to rely on a class that is not available in the class loader: nu/xom/ParsingException
main] i.a.a.c.impl.AxonServerManagedChannel : Requesting connection details from 10.119.119.02:8124
main] i.a.a.c.impl.AxonServerManagedChannel : Successfully connected to 10.119.119.02:8124
main] i.a.a.connector.impl.ControlChannelImpl : Connected instruction stream for context ‘default’. Sending client identification
main] i.a.a.c.command.impl.CommandChannelImpl : CommandChannel for context ‘default’ connected, 0 command handlers registered
main] i.a.a.c.command.impl.CommandChannelImpl : Registered handler for command ‘com.my.test.commonservice.commands.ShipOrderCommand’ in context ‘default’
not explicitly initialized, using predefined black list on your own risk.
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ‘/mypocservice’
main] f.MyPOCSpringbootStarterApplication : Started MyPOCSpringbootStarterApplication in 29.598 seconds (JVM running for 31.813)
1 — [d.api.events]-0] o.a.e.TrackingEventProcessor : Worker assigned to segment Segment[0/0] for processing
1 — [d.api.events]-0] o.a.e.TrackingEventProcessor : Using current Thread for last segment worker: TrackingSegmentWorker{processor=com.my.test.shipmentservice.command.api.events, segment=Segment[0/0]}
1 — [d.api.events]-0] o.a.e.TrackingEventProcessor : Fetched token: null for segment: Segment[0/0]
1 — [nio-8080-exec-2] o.a.c.c.C.[.[.[/jenkinsagentservice] : Initializing Spring DispatcherServlet ‘dispatcherServlet’
1 — [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Initializing Servlet ‘dispatcherServlet’
1 — [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : Completed initialization in 2 ms
1 — [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
1 — [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown : Graceful shutdown complete
1 — [ionShutdownHook] o.a.e.TrackingEventProcessor : Shutdown state set for Processor ‘com.my.test.shipmentservice.command.api.events’.
Thread-1] o.a.e.TrackingEventProcessor : Processor ‘com.my.test.shipmentservice.command.api.events’ awaiting termination…
1 — [d.api.events]-0] o.a.e.TrackingEventProcessor : Released claim
1 — [d.api.events]-0] o.a.e.TrackingEventProcessor : Worker for segment Segment[0/0] stopped.
1 — [ionShutdownHook] i.a.a.c.command.impl.CommandChannelImpl : Unregistered handler for command ‘com.my.test.commonservice.commands.ShipOrderCommand’ in context ‘default’
1 — [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit ‘default’
1 — [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated…
1 — [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.

deepraj · April 23, 2024, 3:15pm

Could someone please help on this as I am not finding any solution available for this issue. I appreciate the help or suggestions in this scenario. Thanks!

brunch · April 25, 2024, 2:27pm

Hi @DeepRaj,

Could you share the EKS file you are using to start this application? I am wondering if you need some level of a liveliness probe to keep this application up?

-Ben

deepraj · April 25, 2024, 3:44pm

Thanks so much Ben for reaching out to me.
Last evening I fixed this issue. The root cause of this issue was that the spring boot actuator jar dependency was missing in my pom.xml and the pod was trying to access the /actuator/health url after start up of tomcat, but as the actuator jar was missing so actuator/health url was not available and pod was bringing down the application then. Once I added this actuator dependency in my pom then tomcat stayed good after starting up successfully.