token store initialization for splitting/merging

Imported from Google Groups
1

Hi,

This might be a dumb question, but I’m trying to make the new split/merge feature work, and run into this:

Currently TEPs are initialized like this:

@Autowired
fun configure(conf: org.axonframework.config.Configuration) = conf.getComponent(TrackingEventProcessorConfiguration::class.java) {
TrackingEventProcessorConfiguration.forParallelProcessing(Runtime.getRuntime().availableProcessors())
}.andBatchSize(eventProcessingBatchSize)

When trying to split segments from the AxonDashboard UI with the plus button, I get: “java.lang.UnsupportedOperationException: TokenStore must require explicit initialization to safely split tokens”

In the Java doc I found:
To be able to split segments, the {@link TokenStore} configured with this processor must use explicitly
initialized tokens. See {@link TokenStore#requiresExplicitSegmentInitialization()}

I’m using mongo token store, extension version is 4.1 (as well as axon and axonserver version of course).

Should I call MongoTokenStore#initializeTokenSegments(String processorName, int segmentCount) or what does ‘explicit segment initialization’ mean?

I’ve tried to drop the tokenstore, but I guess I need to do some configuration, but not quite sure what and how am I supposed to call,
could you please provide an example how do it? Could not find in the reference guide either.

Another thing I noticed, when I try the REST API, I get 500, “No segments found for processor name”
What is the “context” parameter for? Am I supposed to provide something special there to
make it work?

Thanks

2

Hi Vilmos,

Sadly to share this, but the MongoTokenStore wasn’t updated to implement the necessary methods to support split and merge operations.
We did discuss this necessity, but it somehow wasn’t added to our planning for 4.1 in a timely manor.
For that, I give you my sincere apologies, and the following issue on the extension-mongo repository.

I’ve set it to be part of release 4.1.1, which I assume will be released soon.
For now though, all you could do is configure the JPA or JDBC TokenStore, although I assume that’s not an option in your scenario…

Hope this gives you the required guidance.

I’ll ensure to update this post once the feature has been implemented.

In the mean time, thanks for your patience Vilmos.

Cheers,

Steven van Beelen

Axon Framework Lead Developer

AxonIQ

3

Hi Steven,

I see, thanks!

Cheers

4

Hi,

With the 4.1.1 mongo extension, split/merge works from the AxonDashboard UI.
However, I’m still getting 500 “No segments found for processor” when calling the REST API:
/v1/components/{component}/processors/{processor}/segments/split
Am I missing something? What should be the ‘context’ query param’s value?

Thanks

5

Hi Vilmos,

the context parameter reflects the context for which you would like to alter the component. In the standard edition, there is only one context, called “default”.
Most likely, adding ?context=default to your URL will resolve the issue.

Cheers,

6

Hi Allard,

Thank you for the quick help, it indeed solved the issue!

Thanks,
Regards

7

Hi,

With the 4.1.1 mongo extension, split/merge works from the AxonDashboard UI

Well, scratch that, merge still doesn’t work, as it seems.
Test case is simple (everything is 4.1.1: axon, axonserver, axon mongo extension):
I start up a query service, 1 processor, 4 threads/segments.
I want another node of this, so I call split on AxonDashboard on the processor, 4 times to acquire another 4 segments for the new node,
call it one by one, it succeeds. I start up the new node, it claims the 4 new segments, looks fine,
2 nodes with 2 processors, 4 threads/segments each. Then I want to scale down, before shutting down one
of the instances I need to call merge on AxonDashboard, 4 times. However, it does not always succeed.
Sometimes it does, sometimes I get:

Processing merge instruction for segments [3] and [7] in processor [TestProjection]
Failed to merge segment [3] for processor [TestProjection]

Caused by: org.axonframework.eventhandling.tokenstore.UnableToClaimTokenException: Unable to claim token ‘TestProjection[7]’. It has not been initialized yet
at org.axonframework.extensions.mongo.eventsourcing.tokenstore.MongoTokenStore.loadToken(MongoTokenStore.java:182)

Same when calling the REST API, merge fails with this one, very often. But sometimes it succeeds … ?

Another thing, I want to automate this, like make it work with Kubernetes autoscaling, so I need to call the axonserver REST API
(probably from the PostStart/PreStop hooks) before starting/shutting down another node.
It means, for each processor (or at least for those ones that are eligible for scaling up, maybe not all processors of a certain query
service needs scaling, but that is beside the point), I want another n segments for a new node, where n is the thread count of
the tracking processor. It means I need to make a lot of http calls … it would be nice to somehow batch it, and make only one call,
or at least make one call per processor, and have the ability to tell how many new segments I want, or event better, somehow set that
the number of new segments should match the thread count of the processor.
Also what I noticed, this won’t even work with a naive approach, e.g. calling it from a simple loop, because I get this:

o.a.e.TrackingEventProcessor : Processing split instruction for segment [0] in processor [TestProjection]
o.a.a.c.p.EventProcessorController : Successfully split segment [0] of processor [TestProjection]
o.a.a.c.u.FlowControllingStreamObserver : Observer stopped
o.a.e.TrackingEventProcessor : No Worker Launcher active. Using current thread to assign segments.
o.a.a.c.p.EventProcessorController : Was not able to split segment [0] for processor [TestProjection]
o.a.a.c.p.EventProcessorController : Was not able to split segment [0] for processor [TestProjection]
o.a.a.c.p.EventProcessorController : Was not able to split segment [0] for processor [TestProjection]

It looks like it wants to split the same segments 4 times, I need to wait between the calls
to let each split happen first I guess, but I’m not sure how I can tell I can call split again, because it is safe,
the previous split already took place? For the sake of the test, I put a Thread.sleep between the calls, and it is
indeed a matter of timing.
Any suggestions how I could automate and implement this?

Thanks

8

Hi Vilmos,

the operation as it is currently provided, doesn’t assume “high-speed split and merge” usage. Our first priority was the ability to be able to split and merge in the first place. That means that at the moment, you shouldn’t hit the API in quick succession. In fact, that will probably never work, because there is a bit of coordination that needs to be done.

We have some ideas to implement an API where you could provide the desired number of segments, whereafter AxonServer will scale to that number gradually. In that case, you wouldn’t have to worry about coordinating that effort at all.
However, we decided to take it one step at a time, delivering the ability to split and merge, first, and the automated features later.

For now, you would have to put a wait in between. As processors have a backoff cycle of about 5 seconds (by default), waiting for a littlebit longer than that between invocations should suffice.

Cheers,

9

Hi Allard,

I see, thank you, I’ll stick to the wait between approach for now!

Do you happen to have any insights about what I might be doing wrong with the
merge part (from AxonDashboard)? I’m still unable to perform it, I’m wondering what
could be missing.

Thank you

10

Hi Vilmos,

we aren’t aware of any issue with merging other than in 4.1 of the Mongo Extension (which doesn’t support split & merge). Do you see anything in the application logs? Any signs of the instructions coming in (and failing)?

Cheers,

Allard Buijze

11

Hi,

Ok, I’ve realized what I did wrong, I tried to merge segments with all the nodes still running, instead of shutting down one of the nodes first, then call merge - the latter works, I guess that’s the right order.

Thanks,
Regards

12

Hi Vilmos,

it should actually be possible to do that while they are running. I think there is a timing issue with this process in combination with the MongoTokenStore.
We’ll investigate.

Cheers,

Allard