HI Team,
We have issue from the Consumer side when we give an initial segment count more than one. Event handlers missing some of the messages from Kafka. Currently, Kafka has 50 partitions.
Application Yml:
axon:
eventhandling:
processors:
“[kafkaProcessor]”:
source: kafkaMessageSource
mode: TRACKING
threadCount: 1
initialSegmentCount: 1
batchSize: 500
The total number of threads (across all instances of the component) needs to be at least equal to the number of segments.
yes, that’s true. I have given 2 threads per container and 4 segments. We will have at least 2 containers at any time.
i have tried to bring up one instance with 50 segments and 50 threads, but it missed the events. anyone having the same issue?
It seems like missing some message .
I have debugged it by publishing 5 messages and in a single instance of the consumer with 2 segments then while polling eventStream.hasNextAvailable() is returning false and buffer.poll(timeout, unit) returning null after getting 2 or 3 messages. So it missing at least 1 each time.
i am missing any config or is it bug ?
Hi Ranadheer,
Fifty segments and threads is a very, very large amount.
Not trying to be offensive here, but the chances are very slim you actually need that much work going on at the same time.
On the other note, it is good to know that the Kafka extension is not out of beta yet.
Thus, some bugs can be expected. If you find something promising, I would kindly ask you if you can file an issue for it on the Kafka Extension GitHub page.
If you want a more thorough solution fo dispatching events without bugs, I would highly recommend giving Axon Server a try.
That’s my two cents to the situation.
Cheers,
Steven
PS. It is important to note that this mailing list will be discontinued as specified in this thread.
Instead of this mailing list we have moved to use a forum style of communication, at https://discuss-next.axoniq.io/.
Hope to see you there!