Under heavy load, events cannot be persisted to the database

Imported from Google Groups
1

I’m using JMeter to test a heavy load against my system and I am noticing events are failing to get stored in the table under extremely heavy load.

Here is an example of the output that I am seeing:

WARN o.a.commandhandling.callbacks.LoggingCallback - Command resulted in exception: com.ge.energy.markets.bids.demand.eventsourcing.commands.PutAllByDayCommandUpdate
org.axonframework.eventsourcing.eventstore.EventStoreException: An event for aggregate [(day=2050-06-09T00:00-04:00, marketParticipant=Cust 1, pnode=EAST Load Zone 2)] at sequence [81] could not be persisted

It’s at a different sequence every time, but it’s usually after 70-80 events have been persisted rapidly.

I included an image of the thread load that I am running, but I don’t see why the events would not persist, no matter the load I put on the system – as I am running against the same aggregate ID, each and every time.

Screen Shot 2019-02-08 at 8.56.46 AM.png

2

From my testing, it almost looks to me like the next thread kicks off prior

to the event being COMMITTED to the database, causing the database to fail
while trying to submit a sequence which has already been committed

3

Hi Robert,

for my understanding, are you running a single instance, or multiple? Which CommandBus implementation do you use? Which EventStore and Database do you use?

We have seen situations where databases use repeatable read by default, which doesn’t return any information that has been committed while your transaction was running. Changing the isolation level to read committed would help in that case.

Did you consider using AxonServer? This is exactly the type of thing that AxonServer was built to do well out-of-the box.

Cheers,

4

We have a single instance running, utilizing the default CommandBus implementation, out of the box – same goes for the EventStore.

Our database is a PostgreSQL instance 11.1

5

I do have my @Transaction calls, through JPA, to the database as Isolation level of Serializable

6

I also did some research and realized that the Isolation level is set to Read Committed, by default – based on the suggestion that you gave

7

Sorry for all the responses, but Allard, is it possible to set a transaction level on the events being processed within code?

If so, how would I go about doing so? I already have a transaction set from within the command handler which modifies table data – but not the events themselves, being passed to the domainevententry table, where I am seeing the sequence issue.

8

Hi Robert,

are you, by any chance starting a transaction before sending a command to the command bus?

Yes, you can configure the transaction isolation level. If you’re on Spring Boot, just declare a bean of type SpringTransactionManager. It takes Spring’s PlatformTransactionManager and optionally a TransactionDefinition as parameters. The TransactionDefinition allows you to define isolation levels.

Cheers,

9

No, I am starting transactions only from within the command handler itself.

I am calling a service, which then sends a command – inside of the command handler a transaction is executed to modify the database (all of this seems to work fine).

The error I am seeing is related specifically to when the domainevententry cannot be modified due to the sequence number, the error that I showed above.

10

I’m trying to create a SpringTransactionManager, but I don’t see how I set the Isolation level within the bean:

@Bean

public SpringTransactionManager transactionManager(SpringTransactionManager transactionManager)

{

transactionManager.

return transactionmanager;

}

11

As Allard mentioned, it takes a PTM:

@Bean
public SpringTransactionManager springTransactionManager(PlatformTransactionManager platformTransactionManager) {
return new SpringTransactionManager(platformTransactionManager, new DefaultTransactionDefinition());
}

12

Thanks! Sorry, I saw that after I’d already replied, as well.

I assume all events utilize this transaction manager when performing updates against the tables for event storage?

Thanks!

13

Strictly speaking, it’s the Command Handling that uses this transaction manager. Events are also stored while this transaction is active.

Transactions are committed when handling is complete. After that (in the cleanup phase) any locks are released, allowing other threads to access the aggregate again. That’s why it’s important that there is no other transaction manager in front of the command bus, as locks would be released before transactions are committed. In such case, changes may not yet be visible to other threads.

Cheers,

14

There should be no issue with a nested transaction, occurring from within the command handler – is that correct?

That is the only transaction being performed, outside of the command handler itself.

15

Hi Robert,

I’m not entirely sure I understand the structure that you have to store aggregate instances.
Creating separate transactions within a command handler for specific activity should not be a problem, as long as that transaction doesn’t leak out somehow.

In one of your previous emails, you showed your nested @Transactional code. I noticed that the annotation is on a private method. I’m not sure if this is related, but for Spring to proxy annotated methods, they need to be public (unless you use compile/load time weaving with AspectJ).

Are you doing any custom threading or async processing somewhere? Could you provide a brief overview of the flow your commands and events take?

Cheers,

Allard