Missing events in the axon-server

Axon Server
1

Hi,

In one of our projects we are using the axon-server-se (4.4.7). We have currently approximately 640 000 events in the server. However, we are missing approximately 3200 events in the axon-server.

We are creating “external” events based on “local” events in our application (based on an event store local to each micro service) which are then sent to the server. Based on these numbers we know that those 3200 events are missing. Due to our logging we also know that (at least) several of these events have been handled by other micro services. Thus the server would have sent those events to other micro services but failed to properly store them in its binary files.

We would like to know whether this is a know issue? Is it possible that events are forwarded but not properly stored? Would this be different with the enterprise edition?

Thank you and best regards

Nils

2

Hi Nils,
This is not something we have heard of before. The only possible explanation that I can think of is that Axon Server or the server where it was running on crashed before the data could be written to disk, but this is typically something that would cause issues on the next restart.
As you are saying that the events are being processed by other microservices, it means that they have to be written to the file buffer, as the event processors get the events from this buffer. How did you conclude that the events were not in the event store?

In Enterprise Editions there are more guards against write failures, as:

  1. It stores data in a tx-log first, and then processes it into the store
  2. It replicates all data, so if something gets corrupted somewhere (which can happen for a million reasons), you have copies to reconcile from

Regards,
Marc

3

Hi Marc,

Thank you for the quick answer.

We know that the events, once processed in other micro services, cause further processes. According to the analysis of my colleagues, those processes have been triggered, but the events can not be found in the server. Your statement about the data beeing written to disk is very vital. I will inquire whether the logs may have been misinterpreted.

In the meantime we found a bug in the sending component (which transfers the local events to the server) - the corresponding event processor used the default error handler (LoggingErrorHandler instead of the PropagatingErrorHandler). We corrected this bug and will observe whether this solves our issue.

Thank you and best regards

Nils