distributed event implementation

Imported from Google Groups
1

We are currently exploring Axon for our transformation project. So far so good, however we wanted to understand how to address following scenario:

Scenario:

Service1 running on 3 PODs, received a REST request to create payment. Create payment work flow consist of multiple commands and events.

  1. Request received by Service1-POD1, request converted into command (create payment), which creates an event (payment created) and run some projection (query) logic.
  2. “Payment created” event then calls another command (validate payment)

Question 1: if service1-pod1 goes down during step 2 then we will have “payment created” event in event store. How do we ask other PODs like pod2 or pod3 can pick-up the “payment created” event and run step2?

Question 2: what if during projection (query) logic pod goes down, how do we tell other active PODs to pick up the last event but to also run the query handler associated with that event.

Basically, we are looking for distributed event implementation, means once command created an event then any running instance of an application can pick-up that event and process.

2

Hi Shradda,

Excellent questions, fundamental to making something like that reliable.

The key abstraction that Axon offers for this case is the tracking event processor. Every @EventHandler in an Axon program gets assigned to an event processor. This event processor can be configured; details in https://docs.axonframework.org/part-iii-infrastructure-components/event-processing. By default, event processors are ‘subscribing’ but they can be configured to be ‘tracking’.

In a tracking event processor scenario, events are effectively read from an event store, and the processor itself is responsible for keeping track of its location in the event stream, identified by a token. This token should be kept in a TokenStore. If the event processors are all down, or fail to process an event, the token will not be updated. This ensures that events don’t get lost.

An event processor can be deployed in multiple pods. By connecting to a shared TokenStore database, they can co-ordinate their event processing and make sure every event gets processed exactly once. This logic is in Axon Framework.

One way of implementing this, is using a relational database an an EventStore (using the EmbeddedEventStore in Axon). In this case, the tracking will be implemented by polling on the database. Alternatively, you may consider using AxonDB (with or without AxonHub). AxonDB is a combination of a high-performance event store and actively pushing event bus - avoiding the need for polling, and leading to a faster, more elegant solution. Using AxonHub in addition to AxonDB, you would also get a mechanism for distributing commands and queries, and getting real time monitoring of your tracking event processors.

Kind regards,

3

Hi Shradda,

I created some demo code illustration this type of scenario, can be found here: https://github.com/fransvanbuul/paymentdemo

It’s an application that has a rest controller, a payment aggregate, a process to trigger validation after payment has been created, and a query model. The rest controller exposes two method, one to create a payment and one to get the payments. I put swagger on it, so going to http://localhost:8080/swagger-ui.html will let you easily perform some experiments.

You can run 2 or more instances of this application, connected to the same database - see src/main/resources/application.properties for the details of this.

There’s an intentional bug in this application: if the currency of the payment happens to be EUR, it crashes the JVM instantly using Runtime#halt. What you will see is that event processing always happens correctly. If you have 2 instances running, and the one processing the events crashes, another will take over in seconds.

This is standard Axon behaviour (of tracking event processors), in the example case backed by Postgres.

With our commercial offerings AxonHub and AxonDB, you would get a few additional benefits in terms of distribution:

  • You would get a very easy way to distribute commands, events and queries, not just events like in the example.
  • Tracking processors on AxonDB are a lot faster than those on a relational database, because they don’t have to do polling.
    Hope this is useful, I’m available to further discuss/demo as needed.

Kind regards,