AxonServer Exceptions: NullPointerException, BufferOverflowException

Simon_Zambrovski · June 19, 2019, 8:12am

Hi Folks,

we were running AxonServer SE 4.0 as a Docker Image in our OpenShift installation.

Suddenly, the AxonServer stopped working yesterday and started to throw NullPointerExceptions.
After a restart of container, it refuses to start with the following exception in log (see below)

Any ideas?

P.S. Don’t be confused by the start path (jar:file:/home/jboss/opt/axonserver/axonserver.jar), we use RedHat Java OC image for it.

Cheers,

Simon

/ \ __ _____ _ __ / | ___ _ ____ _____ _ __
/ _ \ \ / / _ | ’ \_ \ / _ \ '\ \ / / _ \ '|
/ ___ \ > < () | | | |) | __/ | \ V / __/ |
// _//__/|| ||____/ _|| _/ ___||
Standard Edition Powered by AxonIQ

version: 4.0
2019-06-19 08:03:29.644 INFO 8 — [ main] io.axoniq.axonserver.AxonServer : Starting AxonServer on axonserver-12-wffjc with PID 8 (/home/jboss/opt/axonserver/axonserver.jar started by ? in /home/jboss/opt/axonserver)
2019-06-19 08:03:29.676 INFO 8 — [ main] io.axoniq.axonserver.AxonServer : No active profile set, falling back to default profiles: default
2019-06-19 08:03:44.595 INFO 8 — [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8024 (http)
2019-06-19 08:04:00.913 WARN 8 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘axonServers’ defined in URL [jar:file:/home/jboss/opt/axonserver/axonserver.jar!/BOOT-INF/classes!/io/axoniq/axonserver/rest/svg/mapping/AxonServers.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘eventStoreLocator’: Invocation of init method failed; nested exception is java.nio.BufferOverflowException
2019-06-19 08:04:01.091 ERROR 8 — [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘axonServers’ defined in URL [jar:file:/home/jboss/opt/axonserver/axonserver.jar!/BOOT-INF/classes!/io/axoniq/axonserver/rest/svg/mapping/AxonServers.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘eventStoreLocator’: Invocation of init method failed; nested exception is java.nio.BufferOverflowException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:729) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:192) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1274) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1131) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) ~[spring-context-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at io.axoniq.axonserver.AxonServer.main(y:46) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [axonserver.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [axonserver.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [axonserver.jar:na]
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:593) [axonserver.jar:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘eventStoreLocator’: Invocation of init method failed; nested exception is java.nio.BufferOverflowException
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:138) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1698) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1065) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
… 27 common frames omitted
Caused by: java.nio.BufferOverflowException: null
at java.nio.Buffer.nextPutIndex(Buffer.java:521) ~[na:1.8.0_191]
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:297) ~[na:1.8.0_191]
at io.axoniq.axonserver.localstorage.file.e.d(cd:395) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.e.d(cd:200) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.e.g(cd:89) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.c.d(je:118) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.a.d(nm:453) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.LocalEventStore.initContext(nm:41) ~[classes!/:na]
at io.axoniq.axonserver.z.d.d(lj:46) ~[classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
… 40 common frames omitted

Matthias_Wesolowski · June 19, 2019, 9:04am

Just an idea: do you have enough disk space left?

Simon_Zambrovski · June 19, 2019, 1:35pm

I had the same idea…

The volume was too small, so only 200mb space left. We increased it to become 4GB volume - no changes so far, AxonServer is not starting anymore.

I saved the old volume and re-created a new one - AxonServer started then.

Simon_Zambrovski · June 25, 2019, 1:24pm

So any ideas?

Does it mean that AxonServer breaks unrecoverable if it runs into out-of-space problem once?

Marc_Gathier · June 25, 2019, 1:54pm

Hi Simon,

Could you check in the data directory if there is an *.events file with size 0. AxonServer may have created the file and then when it tried to initialize the buffer afterwards it was not able to do so.
If this is the case you can safely remove that file and restart AxonServer.

Marc Gathier

Simon_Zambrovski · July 1, 2019, 2:29pm

Hi,

sorry for late reply. Thank you - I will definitely try this out.

Kind regards,

Simon