Axon Server cancels long-running streaming queries in cluster

Axon Server
1

Hi,
we have a setup of an Axon Server cluster (2025.2.7) with some applications built upon Axon Framework (4.13.3). For some use cases we’re using Streaming Queries to transport a lot of data between two applications. These queries may have thousands of data records, and a query may take several minutes. Data flow is constantly high with thousands of records per second.

For long-running streaming queries we detected cancellations. Neither the query sender nor the query handler do cancel the query. It seems the Axon Server cancels it as there are two log entries:

  1. WARN io.axoniq.axonserver.message.EnterpriseQueryCache Found 1 waiting queries to delete
  2. WARN io.axoniq.axonserver.message.EnterpriseQueryCache Cancelling query xxx.yyy.zzz.StreamingQuery sent by 1@aaa…, waiting for reply from [1@bbb…]

To keep it simple in this example, assume the following setup: Three Axon Server nodes (1, 2, 3) and two applications (aaa, bbb). Queries are cancelled if both applications are connected two different Axon Servers (e.g. 1-aaa and 2-bbb). If they are connected to the same server, the query won’t get cancelled (e.g. 1-aaa and 1-bbb). Above log statements are logged by the Axon Server which is connected to the application with the query handler. The other Axon server (connected to the app with the query sender) logs nothing.

Cancellation happens after about 5 minutes. Some further investigations showed that the Axon Server property axoniq.axonserver.default-query-timeout controls this timeout (with a default of 5 minutes - there it is).

My questions:

  1. Are Streaming Queries supported in an Axon Server cluster?
  2. What is the idea behind axoniq.axonserver.default-query-timeout? The documentation is brief. It seems it is something like a safe-guard to cancel queries running out-of-control?
  3. What impact has increasing this timeout? What resource requirement has a streaming query on Axon Server?

To meet our business requirements, we’re going to increase the timeout from 5 minutes to 1 hour. The use case is rarely used, we’ll keep monitoring our system.

Best regards,
Matthias

2

Hi Matthias,

Can you give some more information on why the query runs for more than 5 minutes? is it constantly returning results or is it listening to events and then returning a response item when something happens?
If it is the second, subscription queries may be more suited for this.

Axon Server supports streaming queries, but as other queries, Axon Server expects them to complete within 5 minutes (by default). Axon Server cancels queries after the timeout to prevent dead queries remaining in the system forever and free any resources used for the query.

Marc

3

Hi Marc,

The query runs (sometimes) for more than 5 minutes due to the vast amount of data. The query handler is constantly returning results (well, at the beginning there is a delay of a few seconds). That’s why we used a streaming query.

We’re using subscription queries, too, but in this business case a streaming query fits better.

From your answer I conclude that Axon Server supports streaming queries, and there is safeguard which terminates long-running queries after 5 minutes (default setting) to prevent accumulating never-ending queries. That makes sense, and I’m fine with increasing this timeout for my use cases.

It would be great if you added this information to the documentation of Axon Framework (or maybe link to the appropriate Axon Server docs). The timeout setting is documented, but the description is a little bit too brief.

At development time, we used a local Axon Server (single instance, no cluster). Everything worked fine, even queries running for more than 5 minutes weren’t cancelled. But in a cluster, the timeout is applied. The timeout should cancel long-running queries in a single-instance system, too, shouldn’t it?

Best regards,
Matthias