An event processor fell behind the tail end of the event store cache

Imported from Google Groups
1

Hi all,

We’re intermittently seeing these warnings from the EmbeddedEventStore in our prod environment: “An event processor fell behind the tail end of the event store cache. This usually indicates a badly performing event processor.”

This usually happens when there’s very little (or no!) activity in the system. I’ve been able to reproduce the warning locally by adding sleeps to event handlers, but not in a consistent manner.

The initial analysis indicates that this only seems to occur for EventConsumers where lastNode is null and lastToken is a tracking token with some gaps. The lastToken index is comfortably within the range of indexes cached by the EmbeddedEventStore, so I would not expect the tracker to need a private connection to the data store.

Configuration:

  • Spring Boot 2.1.3
  • Axon Framework 3.4.2
  • JdbcEventStorageEngine, JdbcTokenStore, JdbcSagaStore on PostgreSQL
  • Tracking processors for all views and sagas

Not sure if this is a bug, expected behaviour, or a misconfiguration on our part. Any pointers or ideas much appreciated.

Thanks,
Faik

2

The initial analysis indicates that this only seems to occur for EventConsumers where lastNode is null and lastToken is a tracking token with some gaps. The lastToken index is comfortably within the range of indexes cached by the EmbeddedEventStore, so I would not expect the tracker to need a private connection to the data store.

Further investigation has confirmed that this only happens when the event consumer’s lastNode is null. However it does not depend on the existence of gaps in the lastToken tracking token. The nextNode() method is unable to find a matching node when the tracking token has a different configuration of gaps compared to the tokens of the nodes in the cache. Is this by design?

Faik

3

Reported in issue tracker: https://github.com/AxonFramework/AxonFramework/issues/1037

4

Hi Faik,

Thanks again for filing the issue, and sharing it in this thread.
I’d recommend readers who’re encountering the same problem to check the filed issue for further information and suggestions.

Cheers,
Steven