Kafka Auto Commit and Event Processing throughput may result in OOME

Imported from Google Groups
1

When looking at application properties for axon. There does appear to be an option to set auto-commit to false

`

axon:
  kafka:
    consumer:
      enable-auto-commit: false

`

Unfortunately I don’t believe this is currently implemented and supported.
Because our application processes a large volume of messages. Messages are pumped into the memory of the application as quickly as quickly as poll() is called on the consumer api for Kafka.
This can and does result in an OOME under extreme circumstances as the processing of those messages takes far longer than the consumption despite having played with a reduced max-poll-records value of 20 from the default of 500.

I believe the ideal scenario would be to implement manual commits and not grow the internal buffer until the message has either been processed or committed to the event store. Both removing the chance of an OOME and ensuring no messages are lost if the application instance or thread were to crash.

2

Hi Michael,

The provided Spring Boot configuration is, if you check the code, a copied and expanded version from the auto configuration provided by Spring itself for the Kafka stuff.
As you are definitely aware of, the extension is still in a milestone/release candidate state.

I believe you do have found a bug which would be worthwhile to add to the issue list.

As you are quite actively using and testing the extension, we would very much appreciate a contribution from your part here.

What I mean by that is not necessarily the fix itself, but first and foremost the issue with a detailed description like you are giving here.

Cheers,

Steven van Beelen

Axon Framework Lead Developer

AxonIQ
Axon in Action Award 2019 - Nominate your project