One command multiple domain events

We are currently in the process of designing an aggregate as a
"complex aggregate" with entities extending the
AbstractEventSourcedEntity class as described in the docs.
http://www.axonframework.org/docs/1.2/domain-modeling.html#d4e449. For
each command that is processed by the aggregate one or more of the
entities can change. The question is should each change to an
individual entity be a modelled as a seperate DomainEvent or does does
axon need some sort of CompositeDomainEvent that groups a number of
DomainEvents together. The unit of work code seems to support applying
multiple domain events to a aggregate with the a single unit of work.

We have previously implemented an EntityChange concept were a
ComplexDomainEvent would have a collections or EntityChanges instances
associated with it. Each entity could call apply(EntityChange) similar
to how aggregates apply domain events. One drawback with this approach
is that we have to manually unwrap the EntityChange instances from
within the ComplexDomainEvent when they are processed in the
QueryModel or when the aggregate is rebuilt from the event stream.

Hi,

in general, I try not to leak any structural information about the aggregate in my events. Events should express change in functional terms. It is a very common use case to have a single command result in several events, because an aggregate applies changes to several aggregate.

Imaging an Order with Orderlines. If you would have a command “ApplyDiscount”, it is possible that you raise an DiscountAppliedEvent for each OrderLine inside the order.
Alternatively, you could also decide that DiscountApplied is only valid on the Order level (consult your domain expert!). In that case, each OrderLine entity would apply its own discount in the @EventListener method.

Cheers,

Allard

Sheak Said:
"The unit of work code seems to support applying
multiple domain events to a aggregate with the a
single unit of work"
....
"We have previously implemented an EntityChange
concept"

Currently the concept of "The set of domain-events
of a particular aggregate resulted from a single
UOW/high-level-command" is missing in Axon.

I think making this concept explicit does wonders.

Let's call this concept a _Commit_ (following the term
used by Jonathan Oliver of the EventStore fame)
This gives a natural place to persist the meta-data
regarding one UOW/command and grouping the
related events together and make them available
to applications such as Sheak's which may require
them. But more importantly, it makes it possible to
write all the changes to an aggregate in one single
underlying storage's operation. This has many
surprisingly favorable outcomes, without making the
Axon's apps aware of the concept if they don't need
it..

Among others:

It makes two-phase-commit part of history in the
case of relational databases.

It makes supporting no-sql and even file-system
files as the underlying persistent engines a snap
_in a transactional way_. All an engine needs
is the guarantee for atomic single write operation.
which all the prominent ones provide.

It also makes plugging the underlying messaging
systems and asynchronous message distribution
a snap _in a transactional way_.

Off course no wonder comes without a price to
pay. In this case the message consumers must
be able to de-duplicate the received events. But
this is a fact of life in messaging anyways.

It may also have a rippled change effect throughout
the whole Axon. This is why I was hesitant to
bring it up before, but now it is Sheak to blame :slight_smile:

Regards
Esfand

Hi Esfand,

an entry-per-commit approach as you describe will be part of the MongoDB event store in Axon 2. Mainly because of the fact that mongo only has support for transactions within a single document. In a relational database, it could help in the amount of round trips to do, although profit there is not as big.

I don’t see how combining events into a single “commit” can prevent two-phase-commit scenarios. If you want to store to two systems transactionally, it doesn’t matter how many operations you did on each machine.

If you need to be able to correlate events, it is always possible to add meta-data to it. That how other messaging systems do it as well.

Cheers,

Allard

HI Allard,

I think, by mistakem I _replied to author_ instead of _reply_.
I don't have a copy of my message, so please post it back
to the thread if you see fit.

Hi Esfand,

the concept of a durable unit of work (one that can carry on dispatching when a system recovers from a crash) sounds good. One shortcoming of the approach you outlined below is the fact that only domain events are stored in the events store. On the other hand, if they get lost in transit, it is definitely not as bad as with domain events. But a good Durable UoW should be able to recover everything. The only thing that is really unrecoverable is the callback. Clients should not expect a result of the command anymore through the Callback.

In your email, you mention that you need local transactions only, for this process. That’s not entirely true. Your second transaction states:

– The second local transaction pushes the Commit to the
registered event-listeners and (transactionally) marks
the Commit record on the transaction-log as distributed.

There is no way to guarantee that the event listeners and the marking are transactional. If the system crashes right after publishing the events, there is no way to roll that back. That’s only possible if the transaction log is located in the same distribution mechanism as the event listeners. So still, you need a transaction that spans over both these systems.

It’s tough material to discuss over email. But I’m investigation was to make everything more durable than it already is. Keep the ideas coming.

Cheers,

Allard

Hi Allard

Please read in-line:

the concept of a durable unit of work (one that can carry
on dispatching when a system recovers from a crash) ...

To be clear, in the model I'm describing, there is
no "_A_ durable Unit of Work". Instead, the result
of a successful unit-of-work transaction (i.e. the
first local transaction) is the information about one or
more _Commit_ object(s) persisted durably in the
transaction-log. At the end of the first local transaction,
the main transaction is formally considered successfully
committed; no rollback is needed/permitted
passed this point. It is committed but incomplete.
It may remain incomplete indefinitely until all the
local listeners acknowledge the receipt of all the events
unanimously within the same second local transaction
or someone do something about it manually.
I'll come-back to this later but in this model each
and every event-listeners are local/in-process.
Every remote/out-of-process event-listener must
be encapsulated under a plug-in and register as
a local event-listener.

If the system crashes when some of the listeners
have received the events and some others haven't
received the events, the Commit(s) remain
incomplete. The next round all of the listeners
must acknowledge the receipt, even the ones that
had received the same set of events before. Hence
the need for listeners to be idempotent.

If after a crash and before the restart of Axon
someone changes the configuration of the application
and define a modified set of listeners, the new
set of listeners will be the correct one. In
other words a Commit/transaction-log doesn't
contain any information about the recipients.

In this model, the domain-event gets demoted and
is not the main focus of the event-store. Instead
Commits become the focused permanent citizens
of the event-store, each owning a set of domain
events. And yet Commits are hidden from the
event-store's clients (unless the clients express
interest in them), From the clients vantage-point
it is event-store and not commit-store.

The reason I keep using the term _Commit_ object
is because It links this discussion to the
original documentation explaining the Commit concept.

By definition, a Commit contains a set of domain-events that
happened to a given aggregate during a transactional
unit-of-work as well as any meta-data Axon internally
may need to carry it's operations.

One shortcoming of the approach you outlined below
is the fact that only domain events are stored
in the events store.

The transaction-log (and not the event-store) can
contain anything and everything Axon see fit, even
transient information.

A good Durable UoW should be able to recover everything.

Yes. The transaction-log can contain everything needed
to be recovered on a per-Commit level (not a UoW-level).
Which items may be needed to be saved at the UoW-level?
(other than the callback, which is discussed below)

The only thing that is really unrecoverable
is the callback. Clients should not expect a result
of the command anymore through the Callback.

All the listeners are local/in-process listeners. If the
system or Axon process crashes, the next time around, as
part of the system-startup, the callbacks gets registered
again for the current set of local listeners (which may
or may not be the same set before the crash). If there
are remote/external listeners, it it the responsibility
of their in-process local agent/proxy (registered as a
local listener with Axon) to handle the events.

In your email, you mention that you need local transactions
only, for this process.

Yes. That is one of the main goals of this model.

That's not entirely true. Your second transaction states:

-- The second local transaction pushes the Commit to the
  registered event-listeners and (transactionally) marks
  the Commit record on the transaction-log as _distributed_.

There is no way to guarantee that the event listeners
and the marking are transactional.

I think I answer this in the following paragraph. If not
please explain.

If the system crashes right after publishing the events,
there is no way to roll that back.

When the first local transaction is succesfuly committed
(and therefore its entry in the transaction-log is persisted)
the main transaction is considered successfully committed,
even before attempting to call-back any listener or writing it
to the event-store. Off-course it is not completed yet, but it
will eventually become completed. There is no way and no
need to rollback a committed transaction. If the system
crashes in the middle of publishing the event to the local
listeners, i.e. before the successful commit of the second
transaction (which means the transaction-log entry exists
but not marked as _distributed_), after the system resurection,
pushing the events starts all over again. Eventually, all the
listeners will acknowledge together, in the one and
the same local (second) transaction. At that time
pushing the events is considered to be completed, but
still the main transaction is not completed yet -- until
and unless the third transaction successfully commits.
At that time the Commit objects are safe and sound living
in the event-store and the transaction-log is deleted.

That's only possible if the transaction log is located
in the same distribution mechanism as the event listeners.
So still, you need a transaction that spans over both
these systems.

In this model there is no such a thing as
remote/external/foreign etc. Everything and anything
is local. If some remote/foreign/out-of-process systems
are involved (such as a message-oriented-middleware like
ActiveMQ or RabbitMQ) someone must provide an Axon
plugin which looks like a local in-process event-listener
to Axon registered as a local event listener.

This local/in-process event listener can
be provided by Axon out-of-the-box, provided by the MoM
vendor as an Axon plugin, or provided by the application
itself as an Axon plugin. In other words, Axon will have
two main extension-points each declared as an published
Java-interface; one for plugging-in the underlying
persistence engines (SQL or no-SQL) and another for
plugging-in the message listener. All local/in-process
from the stand-point of Axon.

In other words, this model is exactly
the lightweight local transactional event distributor
we are discussing in the other thread, when it comes to
event distribution.

At the end I know many of the above stuff may not be
clear yet. Mainly because at this time we may not have
the same common set of assumptions yet. So, please
don't hesitate to point-out the unclear parts.

Cheers,
Esfand

HI Esfand,

it all sounds very good, I have to admit. Most of the components already work the way you describe. For example, the in-process proxy for event listeners is already there. In fact, that’s the role of the event bus. It’s very easy to implement your own if you have special requirements. And the Event Store (interface) is also already aware of commits, since only a single append() is called for all events of each aggregate.

However, on the other side, some components are missing or in the way. For example, it is the repository which is responsible for storing and publishing domain events. It uses the UnitOfWork for that purpose, so that non-domain events may also be published as part of the same “Transaction”. In Axon 2, all changes made are registered in the UnitOfWork. When that UnitOfWork is committed, it stores the Aggregate and publishes all registered events.

In you scenario’s, you assume that all aggregates are event sourced and that there is only one Event Bus. That assumption doesn’t hold for all applications. Therefore, the durability you describe can only be placed inside the UnitOfWork. It must somehow keep track of which actions have been performed, and which haven’t. Since you cannot assume there is an Event Store, it has to store these somewhere else. On disk, for example. And then, upon recovery, to which Event Bus are you going to publish these events? Which mechanism will have the Event Bus injected?

I was thinking of a scenario where repositories would ask the Event Store on startup if there are any Event/Commits not marked as published. If so, the Repository would dispatch them to the Event Bus that it knows. I am not sure, though, if this will work in a multi-node environment. You need to prevent that events are published multiple times.

The concepts sounds really good, but it needs some more fine-tuning before I am able to implement it.

Cheers,

Allard