Tracking Event Processor + Thread.sleep

Hello,

I’m using a tracking event processor to send out events to another system, but i need to put some delay on some type of events.

I tried to use an easy “Thread.sleep”, but apparently this also blocks any reading consuming of Rabbit via:

axon.amqp.exchange=exchange_out

axon.amqp.transaction-mode=transactional

I can’t understand the “Thread.sleep” will sleep the thread for rabbit consuming, but this thread also uses the “eventbus.publish”.
Maybe it’s related to that? But still, it’s not a subscribing event listener, but a tracking event processor that have the Thread.Sleaap.

Kind regards,

Hi Koen,

I don’t see why it would block the consumer, either. I can’t imagine a blocked transaction will delay any consumers.

Yet, blocking threads should really be a last resort. Isn’t there a way to use a ScheduledExecutorService for this?

One thing isn’t completely clear to me about your setup. Do you have a TrackingProcessor that reads from RabbitMQ? Or do you have a TrackingProcessor that sends messages to amqp as a result of other events (and sleeping for a while), and then another processor to read messages from Rabbit?

Cheers,

Allard

H Allard,

We have a subscribing event listener listening to rabbit (source is rabbit), and pushes these message on the event bus. As the event bus is also an event store, this will result in saving the messages to the event store (domain_event_entry table).
Then we have a tracking event processor, playing all events that are stored in the event store. In this tracking event processor we needed a quick fix, and added Thread.sleep.

Two situations:

  1. The thread sleep blocks the subscribing event listener … maybe it’s a block in the eventbus.publish method. I didn’t find the time to debug this, sorry.
  2. When running 2 instances, the Thread sleep also doesn’t update the token_entry table … and after a timeout, the 2 instances also start processing the same message. I was just wondering if this also can happen when the processing of an event takes for example 20s for some reason (instead of Thread.sleep(20000))

I’ll look into the quart scheduling ATM, to replace the thread sleep.

Kind regards,

Hi Koen,

there was an issue in 3.0 (fixed in 3.0.7, see https://github.com/AxonFramework/AxonFramework/issues/422) where tracking processors would continue processing the same events once processing of an event takes longer than the claim timeout on a token (10 seconds by default). Later versions will put a write-lock on the row with the token, preventing any other processor to take over as long as the token is in active use, even if it takes long to process an event.

Also note that, while the thread is sleeping, a database connection might be holding on to a number of read locks in the events store. Depending on database settings, this may have resulted in a page lock (or worse, a table lock), preventing news rows from being added. We had a similar scenario with one of our customers, caused by MS SQL and Java (ultimately, it wasn’t even Axon related), where we debugged together with a DBA, who analysed query plans for the queries that were executed. It showed that, even though we used proper indices, the database resorted to a table scan (locking all rows), because of an implicit conversion.

Hope this provides a few hints.
Cheers,

Allard