In our production environment we have occasionally very large commands . This can go up to 50000 events. Because we are planning to use axon server (now using oracle) , i tried this in a test project.
I simply used the axon-quick-start (chat) application for that, but instead of posting a single message, 50000 events are posted with a special command like :
@CommandHandler
public void handle(LotsOfMessagesCommand command) {
for (int i = 0; i < 50000; i++) {
apply(new MessagePostedEvent(command.getParticipant(), roomId, command.getMessage() + i));
}
}
LotsOfMessagesCommand is just a copy of the PostMessageCommand
now this gives troubles of 3 kinds :
- you cannot read events in axonserver any more, when you query , the spinning weel keeps spinning
- the event listeners only got about 1000 events
- but worse… after a restart you get the stacktrace below, so the event db ha got corrupted.
Also tried it with 10000 and that is still going ok.
Now I know this is insane large nummer of events, and commands should be split to prevent this, but unfortunately this is not an easy task in our system.
Is this a bug in axonserver? Can the corrupt evenstore be fixed in any way ?
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘axonServers’ defined in URL [jar:file:/Users/gerlo/Documents/develop/git/axon-quick-start/axonserver.jar!/BOOT-INF/classes!/io/axoniq/axonserver/rest/svg/mapping/AxonServers.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘eventStoreLocator’: Invocation of init method failed; nested exception is java.util.NoSuchElementException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:729) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:192) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1274) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1131) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$140/1007653873.getObject(Unknown Source) ~[na:na]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) ~[spring-context-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]
at io.axoniq.axonserver.AxonServer.main(y:46) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [axonserver.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [axonserver.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [axonserver.jar:na]
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:593) [axonserver.jar:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘eventStoreLocator’: Invocation of init method failed; nested exception is java.util.NoSuchElementException
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:138) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1698) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$140/1007653873.getObject(Unknown Source) ~[na:na]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1065) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
… 28 common frames omitted
Caused by: java.util.NoSuchElementException: null
at io.axoniq.axonserver.localstorage.file.h.d(cl:97) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.e.d(cd:57) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.e.g(cd:89) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.file.c.d(je:118) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.a.d(nm:453) ~[classes!/:na]
at io.axoniq.axonserver.localstorage.LocalEventStore.initContext(nm:41) ~[classes!/:na]
at io.axoniq.axonserver.z.d.d(lj:46) ~[classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) ~[spring-beans-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
… 42 common frames omitted