Processor not work with kafka + mongodb

Axon Framework
, , ,
#1

Hello people,

I would like to ask for help because I’m suffering from problems with projections that are not executed.
In our project we decided to use axon + kafka + mongodb being kafka and mongodb that we already use.
What happens is that we have 3 projections of our aggregate in a query project separate from the command project, what happens is that of the 3 projections for the same aggregate sometimes not all are executed, simply nothing is done.
And our query base is totally out of sync, it just doesn’t work, we’ve tried everything, it only works again if we delete the record referring to the processorName of the trackingtokens, no error log is shown.
We are using version 4.5 of kafka extension + 4.5 of mongo + 4.5.3 of axon.
Does anyone have any light for us to help us on how to solve this problem?

Our configuration in the query project is as below:

axon:
event-handling:
processors:
stock-projection:
source: streamableKafkaMessageSource
mode: POOLED
thread-count: 6
batch-size: 20
initial-segment-count: 6
sequencing-policy: sequentialPerAggregatePolicy
notification-projection:
source: streamableKafkaMessageSource
mode: POOLED
thread-count: 6
batch-size: 4
initial-segment-count: 4
sequencing-policy: sequentialPerAggregatePolicy
stock-replay-projection:
source: streamableKafkaMessageSource
mode: POOLED
thread-count: 8
batch-size: 15
initial-segment-count: 4
sequencing-policy: sequentialPerAggregatePolicy
log-projection:
source: streamableKafkaMessageSource
mode: POOLED
thread-count: 4
batch-size: 20
initial-segment-count: 2
serializer:
general: JACKSON
events: JACKSON
messages: JACKSON
kafka:
clientid: stock-query-events-consumer
defaulttopic: stockservice.stock.events
consumer:
enable-auto-commit: true
event-processor-mode: POOLED_STREAMING
bootstrapservers: ${kafka.server}:${kafka.port}
properties:
group-id: “stockservice.stock.events.query-consumer”
auto-offset-reset: earliest
properties:
security.protocol: PLAINTEXT
distributed:
enabled: true

#2

Hi Renato,

There are some improvements in 4.5.4 Kafka extension, that might help a little. Since you are used the streaming event processors, I would advise to set enable-auto-commit to false, since the offsets are stored in an Axon way. Do you have the Mongo Token store configured for this?

Could you maybe check the logs for warnings or errors? If I understand correctly, the main problem is that the projections are update correctly sometimes, and recreating the whole projection is the only way to recover?

#3

Yes, I have mongo store configured.
I updated today’s axon version to 4.5.14 and kafka to 4.5.4 today, removing the enable-auto-commit.
the strange thing is that it works for 2 of the projections and screw on one is random

#4

But nothing in the logs related to the failed projection?

#5

nothing, it just doesn’t work.
I think it has to do with the mongo extension,
I’m not sure how to proceed, thinking about trying to migrate to the axon server because I don’t know how to proceed anymore.
Following the log you see it running the projections, and you see that it skips some, then you go and run it again, there it goes and runs them all.
it’s random