Issue while sending a lot of commands

Axon Server
1

Hi guys,

I’m using Axon/Server 4.4.3.

In certain cases, I need to send a lot of commands in a short time (roughly 20k in a few seconds).

After sending a certain amount of commands, Axon raised a warning that the command cache would be full and that I should increase axoniq.axonserver.command-cache-capacity (12500 is the default) which I set to 100000 then.

After repeating my case, I now get a new warning:

22-10-2020 09:23:27.134 [grpc-default-executor-16] WARN o.a.c.gateway.DefaultCommandGateway.lambda$onResult$0 - Command ‘com.axxelia.axxalon.app.execution.api
.message.ChangeSchedulingStatusCommand’ resulted in org.axonframework.axonserver.connector.command.AxonServerCommandDispatchException([AXONIQ-4003] Failed to add request to queue
1@068e3eb6daa1.e24e99f6-2820-43b8-ac2d-013c5fafa8b6.default)

I’m now implementing a pause after sending 1000 commands. It’s clumsy, but let’s see if that helps.

Anyhow: it seems that once the above warnings occur, that the affected commands get lost! Is that right? If so, I think it would be better to raise an Exception that can be catched… so we can handle the problem.

Second question: as the aggregates that are the target for the command all need to receive that specific command - is there a better pattern than to send 1000’s of single commands?

Thanks,

Markus

2

It seems that command handlers are too slow and can’t catch up. The warning message that the command cache is full means that you should slow down the sending of commands or scale-up axon applications. Are command handlers getting process, can u measure the time needed to execute one command handler?

Let me confirm, you just see a warning in the log and you don’t get an exception, ever?
I’m guessing you are doing some bulk command send at a given point in time?

Also, check this discussion Are there any consequences when command handling takes more time?

3

Thanks.

You’re right - im doing some bulk command send.

As soon as I introduce some pause after sending 1000 commands, I get rid of the warning.

But as I said, it’s just a warning (no Exception) and despite that, it seems the excess commands get lost…

4

are you using “sendAndWait” to send commands or you use a send (completable future)? If u are using “send”, do you have an exceptionally block on your completable future?

5

I’m using send().

I do not have an exception Handler on my block - is there a chance that I will receive an Exception despite the fact that nothing appears in the log?

6

It might be, try using sendAndWait first, and see if u get an exception.

7

Ok, I’ll do.

By the way - the command handler resides in another microservice than the sending one…

8

With sendAndWait it works - no exception.

Thanks a lot!