Parallel processing of multiple tracking event processors

I have three event processors, one is a standard event handler, two are saga event handlers. I was under the impression that they should hand events asynchronously, independent of each other. (i.e. multiple threads running at the same time). However from my debugger, I am observing that the three processors are run one after the other. Is this supposed to be this way? I read in some previous posts, there is no way to control the sequence of saga event handler execution in one processor group. This is not my confusion. The two saga event processors are in different packages. Can someone enlighten me? Thanks.

Hi,

if they are in different packages, they will be assigned to different processing groups, and (by default) also to different processors. In debugging mode, they can be handled sequentially, as the debugger is blocking the JVM’s threads. What happens if you add logging? Do you see different thread names logging the statements?

What does you configuration look like?
Cheers,

Thanks for the response. Yes they are in different packages. One standard event handler: com.xxx.poc.LoggingEventHandler, one saga: com.xxx.poc.payment.PaymentSaga, another saga: com.xxx.poc.shipping.ShippingSaga. I understand they will be in different processing groups. (And I saw three entries in TOKEN_ENTRY table). I did not do any specific configuration as in version 4, default is tracking processor (I verified this because the breakpoint I set on TrackingEventProcessor.start() was hit). Following your suggestion I printed out the logging with Thread.currentThread().getId() in my @EventHandler and @SagaEventHandler, they do have different ids. That proves they do run in different threads. But that does not necessarily mean they are running in parallel (Thread scheduling?)

I use IntelliJ and understand the breakpoint could suspend ALL or just the current THREAD itself. I made sure it is suspended on thread. (I have debugged apps with multiple threads the same time for a long time, IntelliJ debugger will prompt me to switch threads). I would expect to see three threads hitting my different breakpoints once an event is published. This however never happened.

So can someone answer why I am observing that these tracking processors are not running in parallel but sequentially (even though they are all in different threads)? Seems like these threads are being scheduled somehow.

What database do you use? The only thing that I can think of that could cause sequential processing, is if the database doesn’t use row-level locking.

I am using hsql database in my test. I do notice that select * from token_entry table is locked during the processor execution. I will try to select a single row to see if it is locked.

Allard,

I read your reply from the other post, "Event Handlers use a different mechanism. There, it depends on whether you use a Tracking processor, or a subscribing one. In the case of a Tracking Processor, each event will be handled once by any component. There can be concurrent access, but not for the same event. " Since my observation is for the same exact event, your message confirms the behavior I observed. (I will try different events later to verify concurrency) But can you explain why this is the case? Do you put a lock on the event row from domainevententry table? So in the case of Kafka consumer, since there is no event table to lock, even the same event could be handled concurrently by multiple event handlers, correct?

Hi,

It seems that sentence is ambiguous: on a single instance of a processor even if you have multiple threads running, Axon will ensure that the same event is handled by exactly one handler (per class). So if you have a single event handler, and 2 threads, only one of these threads will handle the event.

The only lock in place is on the TokenStore table, where the tracking processor records its progress. To prevent other instances from “stealing” a token while the processor is processing an event, it locks the row containing the token.
If your database doesn’t support row-level locking, which is the case on HSQL, it will lock the entire table instead. This causes other threads to block, waiting until it can locks it own token for processing. As a result, only one thread will process an event at a time.

Try running on H2 instead of HSQLDB, or a full-fledged relational database, such as MySQL or Postgres.

Allard,

Thanks for the reply. “If your database doesn’t support row-level locking, which is the case on HSQL, it will lock the entire table instead. This causes other threads to block, waiting until it can locks it own token for processing. As a result, only one thread will process an event at a time.” I think that is reason.