Tracking event processor log noise with multiple replicas

Imported from Google Groups
#1

Hi,

I’m using tracking event processors in a service I’m currently developing. The service is deployed as (at least) two replicas behind a load balancer.

The replica that comes up first successfully claims a token and starts tracking. However, subsequent processors create a lot of (log) noise because they are unable to claim / release the same token.

The replicas are identical (in terms of configuration). I’d rather not add a flag or setting to mark an individual replica as “special”, and have only that replica start a tracking processor.

Any thoughts how I might handle this?

regards,

Michiel

#2

Hi Michael,

You could solve that by using a Leadership election algorithm. Only the elected leader node should then launch the tracking event processor.
Whenever that node fails another node will become leader.
Leadership electon can be perfomed with Apache Zookeeper.

Cheers,
Benoît

#3

Hi Benoît,

I’ve considered this, but that would lead to the event processor being a subscriber on the non-leader nodes, which is not something I want. I can of course stop tracking event processors on the non-leader nodes as soon as they start, but that doesn’t seem like a nice solution either.

The token-claiming strategy used by axon seems to work fine, it’s just the (unnecessary) log noise that I’d like to reduce :slight_smile:

regards,

Michiel

#4

Michiel,

The purpose of leader election is exactly to prevent non-leader nodes to start their event processor instance.
The leader election algorithm wil appoint exactly one node as leader and only that node should start the tracking event processor. When a node’s leadership is revoked, that node should stop the event processor.

Regards,
Benoît

#5

Hi Benoît,

I understand that, I’m not new to leadership election algorithms.

I’m saying that in default mode, axon 3 will start a subscribing event processor for those event handlers unless I start a tracking event processor. On the non-leader nodes, I want neither, or have a tracking processor sit idle without log noise.

Cheers,

Michiel

#6

Michiel,

some of the log noise was incorrectly generated by Axon. I just “baked” a 3.0.4 release, which resolves much of that. It just logs once saying “Token is owned by another node. Waiting for it to become available…” (on info level). Then it stays quiet.

Cheers,

Allard

#7

Hi Allard,

Awesome, thank you for that!

Regards,

Michiel

#8

Hello. Sorry to rebump this but I think my issue is related. What is the expected behavior when a “Token is owned by another node. Waiting for it to become available…” happens? We have 2 instances and after that info one of the instances starts lagging out of control. What would be the recommended solution to this issue?

Thanks!

#9

Hi Jack,

I am not certain I fully comprehend the issue you are experiencing.

So, I am gonna give some background, which I hope will answer the question you have.
If not, feel free to reply on this with a follow up of course.

What I can share though, is that two different instances of your application, which both have the same TrackingEventProcessor, will compete over a Token.

By default, there is only one segment for a given token, so only one of your instances would be able to use it.
If you want the share the load more equally, you can up the number of initial segments your TrackingEventProcessor uses, as described here.

The class you would be locking for to configure this in Axon, is the TrackingEventProcessorConfiguration.

In there you can specify the number of segments you can use for a given Tracking Event Processor.

I hope this helps you out Jack.

Cheers,
Steven