How does Axon use gRPC streams?

Axon Framework
1

Hello! Thank you for your time.
My question could be seen in the topic, so I want to explain the case where I need implementation gRPC streams in Axon.

I have an application that stores a lot of data, also I have a REST endpoint to create a XLS file based on those data. Internaly I send a Query to my projection to collect data. So, assuming that query is based on a Message, when data is big it could exceed maximum Message size which is 4,2 Mb.

I read about gRPC streams and wonder whether axon uses somehow gRPC streams ? In case I wouldn’t want to manually increase maximum message size in settings.

2

Hello! Thanks again for reading this post.

Any comments on the question above ? :slight_smile:

3

Axon does indeed use gRPC streams to send messages between clients. If your query response exceeds the maximum message size (4MB) you need to customize the maximum message size both for the client applications and for Axon Server.
For the client the property to set is axon.axonserver.max-message-size, for the server the property is axoniq.axonserver.max-message-size. You need to set both, as Axon Server should be able to receive the message from the query handler and the client needs to be able to receive the message from Axon Server.

4

Hello, Marc, thank you for your answer!

My case was to Query some data for building a xls file, and that data handled in a message could be about 100 mb, is it the best way to use axon Query to build such a big file ?

Or there is could be some another way in Axon ?

5

Hi, curious how you solved this. Would it work to have a REST endpoint just for downloading the large file, and send the url as part of the query response?

6

Hi @VladimirTitov4 and @Christopher_Wu, how are you doing?

So, this is my take on the question, hope that helps to clarify a bit :slight_smile:

First of all, if you are exceeding the (default) message size of 4MB, IMO, that indicates something is wrong with how you are using your messages.
For me, messages should be as small as possible and with a flat structure.

Having said that, to your problem of a file generation, I would solve it in a different way:

  • First of all, it sounds like a Command and not a Query, since you are “requesting” something to be build
  • Following this approach, you would have a component responsible to generate such a file (probably in an async way) and just notifies whoever is interested that the file is generated and available at x.y.z
  • In that case, your event would only contain the location where to get the file from (it can be an url, an id or whatever that identify it but not the file itself)
    • There are lots of good storage options for files (S3 for example) but messages are not one of them =)
  • Another component could listen to that event and ‘populate’ a projection with id / filename / location / etc and a query could be made towards that projection to get the info from where the file is
  • Something that we have seen a lot are notification services that will notify the user who requested the file that it is ready to be downloaded “clicking here”. You can send a popup, email or any kind of notification you see fit

That are some ideas of how I would tackle this problem and I hope it helps to shed some light on it!

KR,

1 Like