Reverting back Axon Unit of Work (UOW) on axon 2.4.X

Ajinkya_Virkar · October 8, 2018, 9:16pm

Hello,

I am working on a project built on Axon 2.4.X and I see a behavior, when the EventBus (RabbitMQ) is down. Axon UOW does not roll back the event from the eventstore (MongoDB). I thought Axon will take care of that and remove the event from eventstore if there is some issue while publishing it on the EventBus.

I did go through the 2.4.X code base and noticed that it does revert back the UOW but does not do anything to rollback from EventStore.

I am attaching the logs below.

`

2018-10-08 16:52:58.050 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Registering listener: org.axonframework.repository.LockingRepository$LockCleaningListener
2018-10-08 16:52:58.050 INFO [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] c.u.d.handler.EnvironmentCommandHandler : Finished Handling ProvisionEnvironmentCommand for environment ‘e28fcab3-1a50-40b2-877d-afc20577be51’
2018-10-08 16:52:58.050 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.NestableUnitOfWork : Committing Unit Of Work
2018-10-08 16:54:34.425 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listeners of commit request
2018-10-08 16:54:37.575 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listener [org.axonframework.repository.LockingRepository$LockCleaningListener] of upcoming commit
2018-10-08 16:54:45.315 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Listeners successfully notified
2018-10-08 16:54:49.252 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.DefaultUnitOfWork : Persisting changes to aggregates
2018-10-08 16:54:52.528 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.DefaultUnitOfWork : Persisting changes to [com.ultimatesoftware.domain.aggregate.EnvironmentAggregate], identifier: [e28fcab3-1a50-40b2-877d-afc20577be51]
2018-10-08 16:55:01.370 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.eventstore.mongo.MongoEventStore : 1 events appended
2018-10-08 16:55:18.762 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.DefaultUnitOfWork : Aggregates successfully persisted
2018-10-08 16:56:37.750 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.NestableUnitOfWork : This Unit Of Work is not nested. Finalizing commit…
2018-10-08 16:56:57.089 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.DefaultUnitOfWork : Publishing events to the event bus
2018-10-08 16:57:06.855 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.DefaultUnitOfWork : Publishing event [com.ultimatesoftware.domain.events.EnvironmentProvisionedEvent] to event bus [org.axonframework.eventhandling.ClusteringEventBus@5d2061b7]
2018-10-08 16:57:08.448 ERROR [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] c.ultimatesoftware.aop.ExceptionAspect : exception caught
2018-10-08 16:57:08.457 ERROR [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] c.ultimatesoftware.aop.ExceptionAspect : exception caught
2018-10-08 16:59:09.403 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.NestableUnitOfWork : An error occurred while committing this UnitOfWork. Performing rollback…

org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:58)

2018-10-08 17:00:55.819 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listeners of rollback
2018-10-08 17:00:55.820 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listener [org.axonframework.repository.LockingRepository$LockCleaningListener] of rollback
2018-10-08 17:01:09.949 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.NestableUnitOfWork : Stopping Unit Of Work
2018-10-08 17:01:23.843 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listeners of cleanup
2018-10-08 17:01:23.844 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Notifying listener [org.axonframework.repository.LockingRepository$LockCleaningListener] of cleanup
2018-10-08 17:01:23.845 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.u.UnitOfWorkListenerCollection : Listeners successfully notified
2018-10-08 17:01:27.747 DEBUG [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] o.a.unitofwork.NestableUnitOfWork : Clearing resources of this Unit Of Work.
2018-10-08 17:02:19.227 ERROR [bootstrap,1fcf5b43215af740,30e7fa8e46847a0b,false] 10012 — [tp2052572633-31] c.ultimatesoftware.aop.ExceptionAspect : exception caught
2018-10-08 17:02:22.547 ERROR [bootstrap,1fcf5b43215af740,1fcf5b43215af740,false] 10012 — [tp2052572633-31] c.ultimatesoftware.aop.ExceptionAspect : exception caught
2018-10-08 17:02:22.557 ERROR [bootstrap,1fcf5b43215af740,1fcf5b43215af740,false] 10012 — [tp2052572633-31] c.u.service.config.RestExceptionHandler : A CommandExecutionException was thrown in the request

`

allardbz · October 10, 2018, 2:47pm

Hi,

Axon will roll back the Unit of Work, but Mongo doesn’t (at least, not at the time) support transactions. Events are physically sent in the prepare-commit phase. If something goes wrong after that, and there is no transaction to fall back on, there is not much Axon can do.

Cheers,

Allard

Ajinkya_Virkar · October 11, 2018, 4:02pm

Hello Allard,

Thanks for the useful insight. I was able to dig a little bit deeper on the 2.4.X codebase and was curious about

DefaultUnitOfWork.class
How exactly does backing transaction work?
And what does rollback really do here?
Is it possible to use backing transactions for handling mongo transactions.

@SuppressWarnings("unchecked")
@Override
protected void doRollback(Throwable cause) {
    registeredAggregates.clear();
    eventsToPublish.clear();
    try {
        if (backingTransaction != null) {
            transactionManager.rollbackTransaction(backingTransaction);
        }
    } finally {
        notifyListenersRollback(cause);
    }
}

@SuppressWarnings("unchecked")
@Override
protected void doCommit() {
    do {
        publishEvents();
        commitInnerUnitOfWork();
    } while (!this.eventsToPublish.isEmpty());
    if (isTransactional()) {
        notifyListenersPrepareTransactionCommit(backingTransaction);
        transactionManager.commitTransaction(backingTransaction);
    }
    notifyListenersAfterCommit();
}

EventPublisher.class
Does disruptor command bus solve the issue related to commits?
I see here that the EventStore and EventBus are used in the same method and it might be possible to rollback?

private void storeAndPublish(DisruptorUnitOfWork unitOfWork) {
    DomainEventStream eventsToStore = unitOfWork.getEventsToStore();
    if (eventsToStore.hasNext()) {
        eventStore.appendEvents(unitOfWork.getAggregateType(), eventsToStore);
    }
    List<EventMessage> eventMessages = unitOfWork.getEventsToPublish();
    EventMessage[] eventsToPublish = eventMessages.toArray(new EventMessage[eventMessages.size()]);
    if (eventBus != null && eventsToPublish.length > 0) {
        eventBus.publish(eventsToPublish);
    }
}

allardbz · October 26, 2018, 9:05am

Hi,

DefaultUnitOfWork.class
How exactly does backing transaction work?
There are handlers that hook into the unit of work lifecycle that manage the transaction. The UoW itself doesn’t handle any transactions. It merely tells a TransactionManager to start, commit and rollback at specific times.
And what does rollback really do here?
It just invokes rollback handlers and tells a transaction manager to roll back. Any staged actions (such as saving aggregates) are cancelled/ignored.
Is it possible to use backing transactions for handling mongo transactions.

Of course. It does require the MongoEventStore to use a slightly different API, though. The current implementation doesn’t do that.

EventPublisher.class
Does disruptor command bus solve the issue related to commits?
I wouldn’t look for the solution of the Event Store commit problem in the Command Bus implementation. The problem is just that the MongoEventStore doesn’t participate in transactions.
I see here that the EventStore and EventBus are used in the same method and it might be possible to rollback?

All command buses basically first store their aggregates and then publish events. This is nothing specific to the DisruptorCommandBus.

Hope this clarifies things a bit.
I heavily recommend upgrading to more recent versions of Axon. It seems you’re addressing problems that have long been solved in more recent versions.

Cheers,

Allard