Use MySql for EventStore and MongoDb for Projection - Events recreating records

Axon Server
, ,
1

Hi,
I have an existing app that saved collections in MongoDb. As mentioned in documentation and on Event Storage in Axon Server - How does it work? • Allard Buijze • GOTO 2019 - YouTube that MongoDB is not right and that I should use plain RDBMS like MySQL /- Axon Server.

I am able to do that but when I restart the server, all the events are replayed and I get collections repeated again.

Here’s how my code looks like.

Below, I am using MongoDb for saving Products in ProductDB and MySQL for eventStore.

Application Yaml

spring:
  data:
    mongodb:
      database: productsDB
      port: 27017
      host: localhost
  mvc:
    pathmatch:
      matching-strategy: ant_path_matcher
  datasource:
    url: jdbc:mysql://localhost:3306/mysql?autoReconnect=true&useSSL=false&allowPublicKeyRetrieval=true
    username: dinesh
    password: abc123
    name: mysql
  jpa:
    show-sql: true
    hibernate:
      ddl-auto: create
    properties:
      hibernate:
        dialect: org.hibernate.dialect.MySQL8Dialect

##Axon configuration
axon:
  serializer:
    events: jackson
    general: jackson
    messages: jackson

In Pom.Xml, I tried excluding axon-server-connector and including as well.

<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-mongodb</artifactId>
            <version>2.7.6</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>2.7.6</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-rest</artifactId>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.projectreactor</groupId>
            <artifactId>reactor-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-boot-starter</artifactId>
            <version>${springfox.version}</version>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger-ui</artifactId>
            <version>${springfox.version}</version>
        </dependency>
        <dependency>
            <groupId>org.axonframework</groupId>
            <artifactId>axon-spring-boot-starter</artifactId>
            <version>${axon-spring-boot-starter.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.axonframework</groupId>
                    <artifactId>axon-server-connector</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>commons-beanutils</groupId>
            <artifactId>commons-beanutils</artifactId>
            <version>${commons-beanutils.version}</version>
        </dependency>

        <dependency>
            <groupId>org.axonframework.extensions.mongo</groupId>
            <artifactId>axon-mongo</artifactId>
            <version>${axon-mongo.version}</version>
        </dependency>
        <dependency>
            <groupId>org.axonframework</groupId>
            <artifactId>axon-metrics</artifactId>
            <version>${axon-metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>com.mysql</groupId>
            <artifactId>mysql-connector-j</artifactId>
            <version>8.0.31</version>
        </dependency>

Here’s my AxonConfig


@Configuration
public class AxonConfig {

    @Bean
    public EntityManagerProvider getEntityManagerProvider(){
        return new ContainerManagedEntityManagerProvider();
    }


   /* @Bean
    public EventStore eventStore(EventStorageEngine storageEngine, GlobalMetricRegistry metricRegistry) {

        return EmbeddedEventStore.builder()
                .storageEngine(storageEngine)
                .messageMonitor(metricRegistry
                        .registerEventBus("eventStore"))

                .build();
    }*/
   @Bean
   public TokenStore tokenStore( EntityManagerProvider entityManagerProvider) {
       return JpaTokenStore.builder().entityManagerProvider(entityManagerProvider)
               .serializer(JacksonSerializer.defaultSerializer()).build();
   }



    @Bean
    public EventStorageEngine eventStorageEngine(Serializer serializer,
                                                 PersistenceExceptionResolver persistenceExceptionResolver,
                                                 @Qualifier("eventSerializer") Serializer eventSerializer,
                                                 EntityManagerProvider entityManagerProvider,
                                                 TransactionManager transactionManager) throws SQLException {

        JpaEventStorageEngine eventStorageEngine = JpaEventStorageEngine.builder()
                .snapshotSerializer(serializer)
                .persistenceExceptionResolver(persistenceExceptionResolver)
                .eventSerializer(serializer)
                .entityManagerProvider(entityManagerProvider)
                .transactionManager(transactionManager)
                .build();

        return eventStorageEngine;
    }

    @Bean
    public EventUpcasterChain eventUpcasters(){
        return new EventUpcasterChain();
    }


}

I start my axon server and Application.

Step 1: Http POST Create a new product (WORKS)
Step 2: On Axon Server dashboard , event is created (WORKS)
Step 3: In MongoDb, Product collection is created and entry can be verified (WORKS)

Step 4: Stop the Springboot applicatio and restart.
Step 5: MongoDb has an additional record because event was played again.

What’s missing here?
Here’s what I already tried:

  1. Create EventStore also in MOngo (Works) but I don’t want to use Mongo for eventStore and want to stick to MySql/PostgreSql for Axon server ( All examples on Youtube, documents just show with H2)

  2. Tried adding and also removing the axon-server-connector (DomainEventEntry is not created in either case)

  3. When I enable the code to exclude

 <exclusions>
                <exclusion>
                    <groupId>org.axonframework</groupId>
                    <artifactId>axon-server-connector</artifactId>
                </exclusion>
            </exclusions>

I get below error

2022-12-03 03:24:47.844  WARN 43476 --- [cessor[event]-0] o.a.e.TrackingEventProcessor             : Fetch Segments for Processor 'event' failed: org.hibernate.hql.internal.ast.QuerySyntaxException: DomainEventEntry is not mapped [SELECT MIN(e.globalIndex) - 1 FROM DomainEventEntry e]. Preparing for retry in 1s

java.lang.IllegalArgumentException: org.hibernate.hql.internal.ast.QuerySyntaxException: DomainEventEntry is not mapped [SELECT MIN(e.globalIndex) - 1 FROM DomainEventEntry e]
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:138) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:188) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:757) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:848) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:114) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:311) ~[spring-orm-5.3.24.jar:5.3.24]
	at com.sun.proxy.$Proxy147.createQuery(Unknown Source) ~[na:na]
	at org.axonframework.eventsourcing.eventstore.jpa.JpaEventStorageEngine.createTailToken(JpaEventStorageEngine.java:361) ~[axon-eventsourcing-4.6.2.jar:4.6.2]
	at org.axonframework.eventsourcing.eventstore.AbstractEventStore.createTailToken(AbstractEventStore.java:171) ~[axon-eventsourcing-4.6.2.jar:4.6.2]
	at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.lambda$run$1(TrackingEventProcessor.java:1218) ~[axon-messaging-4.6.2.jar:4.6.2]
	at org.axonframework.common.transaction.TransactionManager.executeInTransaction(TransactionManager.java:47) ~[axon-messaging-4.6.2.jar:4.6.2]
	at org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1216) ~[axon-messaging-4.6.2.jar:4.6.2]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.hibernate.hql.internal.ast.QuerySyntaxException: DomainEventEntry is not mapped [SELECT MIN(e.globalIndex) - 1 FROM DomainEventEntry e]
	at org.hibernate.hql.internal.ast.QuerySyntaxException.generateQueryException(QuerySyntaxException.java:79) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.QueryException.wrapWithQueryString(QueryException.java:103) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:220) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:144) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:113) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:73) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.engine.query.spi.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:162) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.AbstractSharedSessionContract.getQueryPlan(AbstractSharedSessionContract.java:636) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:748) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	... 14 common frames omitted
Caused by: org.hibernate.hql.internal.ast.QuerySyntaxException: DomainEventEntry is not mapped
	at org.hibernate.hql.internal.ast.util.SessionFactoryHelper.requireClassPersister(SessionFactoryHelper.java:170) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.tree.FromElementFactory.addFromElement(FromElementFactory.java:91) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.tree.FromClause.addFromElement(FromClause.java:77) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.HqlSqlWalker.createFromElement(HqlSqlWalker.java:334) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromElement(HqlSqlBaseWalker.java:3782) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromElementList(HqlSqlBaseWalker.java:3671) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromClause(HqlSqlBaseWalker.java:746) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.query(HqlSqlBaseWalker.java:602) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectStatement(HqlSqlBaseWalker.java:339) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.statement(HqlSqlBaseWalker.java:287) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.analyze(QueryTranslatorImpl.java:276) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:192) ~[hibernate-core-5.6.14.Final.jar:5.6.14.Final]
	... 20 common frames omitted

What else can I try?

2

I figured it out what was happening. I think this will help people who are getting same issue.

Summary:
Scenario 1
By default when you use

<dependency>
            <groupId>org.axonframework</groupId>
            <artifactId>axon-spring-boot-starter</artifactId>
            <version>${axon-spring-boot-starter.version}</version>
 </dependency>

the axon-server is included and app the wants to use the axon-server to store events. The Axon framework, will pick the first available JPA from you code and use it to store the events.
Ex: If you use “JUST” H2 or MySQL database, the “Your tables” along with Axon-server event tables like Saga/Token Entry etc are saved in the same schema. This is when you don’t create any AxonConfiguration and override to create a different Schema.

Senario2:
You have a table called “Product” in schema “ProductDB”. On an event handler senario, your data will get saved in ProductDB/Product table. If you have not added any Axon configuration class, AxonAutoConfigure will save the Axon-server table also in “ProductsDB” ex: ProductsDB/Token-entry.

If you want to keep Same DB but different Schemas ex: Products in ProductDB and Axon tables in AXONDB schema then you need to write AxonConfiguration, provide EntityManagerProvider, TokenStore and EventStorageEngine.

Scenario 3
This was my scenario. My entity Product is saved in MONGODB and I wanted to save Axon events in MySQL. Now I added Axon configuration and provide the three amigo beans

@Bean
    public EntityManagerProvider getEntityManagerProvider(){
        return new ContainerManagedEntityManagerProvider();
    }

   @Bean
   public TokenStore tokenStore(EntityManagerProvider entityManagerProvider) {
       return JpaTokenStore.builder().entityManagerProvider(entityManagerProvider)
               .serializer(JacksonSerializer.defaultSerializer()).build();
   }

@Bean
    public EventStorageEngine eventStorageEngine(Serializer serializer,
                                                 PersistenceExceptionResolver persistenceExceptionResolver,
                                                 @Qualifier("eventSerializer") Serializer eventSerializer,
                                                 EntityManagerProvider entityManagerProvider,
                                                 TransactionManager transactionManager) throws SQLException {

        JpaEventStorageEngine eventStorageEngine = JpaEventStorageEngine.builder()
                .snapshotSerializer(serializer)
                .persistenceExceptionResolver(persistenceExceptionResolver)
                .eventSerializer(serializer)
                .entityManagerProvider(entityManagerProvider)
                .transactionManager(transactionManager)
                .build();

        return eventStorageEngine;
    }

Now the problem as I stated was that everything was fine, but in MySQL, I was not seeing domain_entry_event table and snapshot_entry_event tables. I assumed it was OK as I was using AXON-SERVER. The problem came when I restarted the application and I saw that Events were getting reapplied which was already executed to save Product entry in DB.

Root Cause: When you use a Jpa then you must also override default configuration to tell axon to stop using Axon-server (exlude axon-server-connector from starter), then you must tell the persistent context to scan table
@EntityScan(basePackageClasses = {DomainEventEntry.class, SagaEntry.class, TokenEntry.class})

Addinng Entity Scan fixed the issue

@SpringBootApplication
@EnableSwagger2
**@EntityScan(basePackageClasses = {DomainEventEntry.class, SagaEntry.class, TokenEntry.class})**
public class EventEventApplication {

    public static void main(String[] args) {
        SpringApplication.run(EventEventApplication.class, args);
    }

}
3

This works, but I think having the saga’s and token store still in MongoDB would be preferred? It doesn’t matter to much, but since the projections would be still in MongoDB, that means all the event processor/query model things would be in MongoDB.

4

@Gerard So initially that is what I did, everything in Mongo but then on this forum and as it says in documentation that RDBS is preferred, I switched everything Axon to MySQl. I was reading several posts on SO and here on this forum as well that people ran into issues with Saga when using Mongo. Hence decided to stick to MySql.

To answer your question yes, projections are all in NoSql Mongo. I cannot change that as it is an existing service.

5

Not everything, and indeed the event storage engine would indeed be better with Axon Server, or RDBS if that’s not an option. Still indeed preferred over MongoDB.

However now you have the Token Store and the projection split, it’s more likely your projection becomes inconsistent. If there is a problem with MongoDB, but not with MySQL, it might update the token store, while the projection was not updates.

Currently consistency with MongoDB is also an issue, since it doesn’t support transactions. So even with MongoDB the projection might be updated, while writing to the token store fails, which doesn’t revert the change to the token store. This will change with the 4.7.0 release through.

About the problem with Mongo and the saga store here, AFAIK they all had to do with serializers, and not specifically with the MongoDB implementation of the saga store. And the same thing as for the projection applies here. For consistency it’s better to store the saga’s and the tokens in the same system.

1 Like