Best practice for Event Sourcing events querying from Aggregate

A design model for parallel model (fowler)http://martinfowler.com/eaaDev/ParallelModel.html

discussed Event Sourcing for processing/querying/replaying of event sourced events.

How would something like this be possible in Axon,

Scenario 1) where when you are calling a domain method, you need to review the event log, and find which event generated what state

e.g. you have a Map of items in a collection in the domain aggregate and every new domain event adds an entry to the collection. now you need to find out which item in the collection was generated by which event… how would you do that from within an aggregate method in Axon?

what would be the simplest way to do this querying from within the aggregate in Axon, any best practice suggestion?

Scenario 2) Creating the parallel model by playing few events, skipping few, applying different events in memory evaluate in a different model to compute a result also seems very powerful to manage state and state correction in the future based on errors in past, however how to dynamically generate that with Axon inside an aggregate or in a command handler is also another best practice question?

Cheers…

To further clarify the axon need, which seems tricky at this time based on some research of the code… but I could be missing something.

-the feature needed would be to get a read only view of all the events of the event store for an aggregate (minus the snapshot events).
eventStore.readEvents does provide this capability, as it only builds from snapshot, there is no way to filter snapshot events so disabling snapshotting for that aggregate may be a way to bypass it but make it less efficient. Also the eventVisitor seems better for migration of the events as it scans all the events in repository.

so e.g. here is a conceptual flow for clarification which could work… but need some input…

  • Pass eventStore to via command handler to the Aggregate method (instead of injecting in Aggregate)

  • Create a DomainEventStream object in the aggregate method and call readEvents on event Store to get the Event Stream for that aggregate(loop through it to search for whatever business linkage need is required, e.g. finding the entry for event matching the addition of a certain item in aggregate collection etc.

  • Create pojo temporary Aggregate with new in aggregate method… in memory… apply handleevents by typeofEvents event handler methods in a sequence desired by business logic by skipping certain events of certain type, applying other hypothetical events etc to rebuild a parallel state (in hindsight state, primarily for error correction or adjustment scenarios for business state) for the business logic and then compute parallel state, deduce business logic for any changes to real domain aggregate and publish another event if needed to adjust original state or publish another event for other external consumption.

what do you think?

This could allow to leverage all the domain model patterns for complex business transactions for enterprise resource planning, financial and other complex applications (reduce complexity) to leverage parallel model described by fowler in his reference and its different usage scenarios and his writeup of EventSourcing and related pattern usage below.

http://martinfowler.com/eaaDev/EventSourcing.html

Your thoughts appreciated…

Cheers…

Hi,

reading through Martin Fowler’s article, I’d expect your scenario 2 to be used. Martin talks about creating a new instance of the/an application which is fed (part of the) events from the original application.

In fact, I am already doing a similar thing for one of my clients. We have gathered many events in the past two years. Now, they are building a number of management reports which are fed from the events. We need to be able to show progress of certain values in time, which means we need to be able to reproduce those events.

One thing you could do is simply visit events and skip those that do not interest you. Not the optimal way, but it will definitely work. You just need to reconstruct a new query model based on those events.
If you want to reconstruct the state of an application, feed those events to the query model AND a copy of the event store which that instance of the application uses. It will allow you to check out the behavior of the application in that state. Martin calls this a “parallel system”.

In Axon 2.0 (snapshot), there is a more flexible query mechanism to visit only a specific range of events. You can use the indexes (the fields stored next to the serialized event) to filter the events that you want returned. It works for both the JPA and the Mongo event store. With this infrastructure, it shouldn’t be too hard to create a parallel system and feed it with a specific selection of events.

Cheers,

Allard

Thanks,

One difference in the parallel model is that for calculation type logic, really want to create a parallel model of aggregate and not all the external consumption of events from the aggregate all the way to the query model. Hence replaying all the way via bus to all end points is not desired.

How could something like that be acheived.

I Like the new EventVisitor code with criteria, which should get a good criteria based event stack, then while looping through events, how to construct parallel desired state e.g. to compute a balance without a certain set of events and some new ones for error correction to determine new balance and then compute the delta to be applied to the original aggregate as an adjustment event… this is another fowler pattern… leveraging parallel model.

http://martinfowler.com/eaaDev/DifferenceAdjustment.html

I think a pojo domain object with applying event handler methods without the bus or axon handling and without the query model and related complexity should suffice?

What do you think?

Also, is 2.0 snapshot stable? Any guidance on breaking changes upgrade from 1.1.2?

Hi,

at the basis of the parallel model is the fact that you have a copy of the application with another data source. So if you want to calculate some “what if” conditions, you’ll need to create a copy of the event store (and optionally query model). The event store copy is fairly simple. Just query all you need from the source, and append in the target.

Copying the query models can be a bit harder (and longer). One way is to just replay the same selection of events on the query models. It may take a while, but you’ll have the correct data. You don’t need to send the events to the bus. You can just send them to the handlers that reconstruct the query models.

I have plans to include some form of “replaying” in Axon 2.0. A replay can be triggered through the Management interface (JMX at first, pluggable later). Listeners must be marked as “replayable”.

Axon 2.0 is not stable yet and contains some breaking changes. When 2.0 is stabilized, I will provide some guidelines and tools to support migration from 1.2. One of the biggest changes is the way events are stored in the Event Store. Meta Data and payload are separated in order to make deserialization (and upcasting) a lot faster.

If you have any questions up-front about the migration, or any concerns, let me know. I’ll try to address them as early in the process as possible.

Cheers,

Allard

Questions on replay and event store appending?

The complexity to deal with event store and datastore seems too high and lot of more work for simple challenge, which I think Axon is excellently positioned for.

The business need for a domain aggregate is something much more simpler

  1. query event log of an aggregate root for a range of events (without snapshot events)
    something like
    DomainEventStream queryEventLog(…) on aggregate root for classes which extend AbstractAnnotatedAggregateRoot or Entity Aggregates

if the query Method with criteria cannot be granular enough to filter the events based on events field information allow for further creation of filtered stream in memory i.e.

DomainEventStream temporaryStream.remove(DomainEvent unwantedCouldNotQueryEvent)

  1. replay on a new in-memory new temporary aggregate (no eventstore or db needed or datasource needed with transactions and commits) to build a what if state.
    somethink like …
    AggregateRoot temporaryAR = new AggregateRoot(String temporaryId);
    temporaryAR.replay(DomainEventStream temporaryStream);

and temporaryAR.getXXXX() for reading resulting state from the AR to compute resluting state and then use that information for error correction or what if business logic, however discard the temporary AR from memory instantly after calculations.

Would this be possible with the design enhancements you are envisioning. It would fulfull a lot of needs of the Accounting patterns discussed by fowler above vs building a full fledged parallel system with datasource transactions, persistence etc, which may also be good additional features for usage.

If this cannot be done by Axon, the other alternative is to keep a collection of eventLog within aggregate as a field append the events on the event handlers and build custom query methods for business domain, create a new pojo Aggregate root in memory e.g.

public class MyAR extends AbstractAnnotatedAggregateRoot{

private CustomCollection eventLog;
private StateField stateField1;
private StateField stateField2;

public MyAR(Stting Id){
}

public doWhatIfOrInHindsightMethod(…){
Collection stream = eventLog.queryByCustomCriteria()
MyAr tempAr = new MyAr(tempUUID);
for (each itemEvent in stream)
{
tempAr.handle(itemEvent);
}
tempAr.getResultantState();

//do more logic and apply any other events as necessary for regular processing and applycation of correction state deduced from the in-memory parallel model.

}

@EventHandler
public handle(Event1 event){
//update state
eventLog.add(event, whenhappened, whenNoticed);
}

@EventHandler
public handle(Event2 event){
//update state
eventLog.add(event, whenhappened, whenNoticed);
}
}

but that would defeat the purpose of the event sourced log and create duplication of event information in log and on the aggregate.

Please give this your thoughts and input/Consideration for future.

Cheers…

One more thing… I missed

Also, in certain situations deling with date ranges Snapshot events would also be included to not go back too much in history if not needed…

Cheers…

After some more thought… to clarify this further… too build accurate state of an object as of a certain date and then apply what if events… you would have to go back to the beginning and replay everything to a certain point in chronology and then apply the what if events to create a parallel model,

hence my assumption above for querying a few events stands corrected.

e.g. if you have to adjust usage of a bill for last month and need a parallel model to calculate, you don’t need to replay every usage event from the beginning of time, you can go to state of the aggregate as of the date for correcting the error.

e.g. you had 50 watts usage on a electicity bill for jan 20-25th reading and it was an error and you got a bill for it when it should have been 30watts for the usage of same period.

  • you only need to get aggregate state as of jan 20th, and for that a snapshot event before Jan 20th and replay to the 20th could get you that state, instead of going to the begining of time e.g. if you were utilitiy customer for ten 10yrs, you would have to replay everything, so to do that in a eventLog field would be harder and non performant in the code explained above, unless there are very small event logs for aggregates, so I guess need Axon feature more than I thought so earlier.

Cheers…

Hi,

I am a bit confused now :wink:

Anyway, replaying events only on an aggregate, just to what state it is in, is not really CQRS. If you want to see state, you need to replay a query model. If you want to verify certain behavior, you replay on an aggregate.
But admittedly, it is sometimes useful to see what state the aggregate is in, just to be able to see how it thinks about its current state.

Work is being done on an Event Viewer, an application that you can install and point to an Event Store to see what’s inside. It might be a nice feature to have a look at aggregate state via that application. I’ll discuss that with the developer.

As to what parallel models is concerned, Axon will provide support for easy selection of events from the store. You could then append all these events to another Event Store instance, which is used by a copy of the application (the parallel system). If you want to test “what if” scenario’s, you’d add some more events to the store now. Then, the application must “replay” all the events on all of its Query Model building Listeners (using Axon provided features). If you now start the application, it is in you “what if” state.

Cheers,

Allard

Sorry for confusion…

but the key difference here is to see the state of the aggregate in what if state vs the state of the entire application… I think thats where I am not communicating very clearly.

… if you read the fowler articles on these patterns for complex error correction scenarios in account oriented and accounting transaction, entry based account or aggregation / summarization domain for any industry, a key principle of rebuilding an aggregates state without the full applications what if logic is what he referes to… as the application could have multiple bounded contexts and you really do not need to go across aggregates to rebuild what if state.

Hope this helps clarify further, if you need me to distill more examples… let me know and I can try to elaborate further from my learnings of the fowler files on this subject. What I have realized in the research is that it is a wonderful feature of event sourcing to manage something of very high complexity by applying pure OO paterns with polymorphism and the power of the event log to regenerate this “in hindsight” or “error corrrection” processing for something which would be very convoluted and highly complex in a non event sourced world. Also, greg young talks about this in his talks on trading floor domain and this is not about full CQRS as much as it is the power of the application of events from the event log to build dynamic state for an object and the chronological advantage of the log. They comparison of this is to version control systems and similar capabilities with branches and merges with deltas(I must have really confused you by now :-))

http://martinfowler.com/eaaDev/DifferenceAdjustment.html

has a very good example of the usage account and shadow account to build the parallel model for the account and its entries collection in the domain object and applying event sourcing to simplify something of a very complex nature in traditional pure OO world. These analysis patterns are fairly elaborate but do require deeper diving to get your head wrapped around them.

Hope to clarify and not confuse. The light bulb for me was how powerful the event log can be too handle this very complex domains of the past and seems very exiting for building very cool apps without the complexity of the past.

This
http://martinfowler.com/apsupp/accounting.pdf

is a excellent document summarizing a lot of the patterns.

The accounting word is misleading at this is really aimed at solving variety of industry problems with our without money.

however fowlers website has most of it in the

http://martinfowler.com/eaaDev/

Events and Accounting patterns sections

As I said, don’t expect anyone to get their head wrapped around the power of these until deeper diving and connecting these designs with the event sourcing universe for event enabled systems, a natural feature of the CQRS universe and feel that Axon is getting well positioned to lead and solve very complex enterprise application domain problems with this super cool infrastructure.

Cheers… and thanks for your patience and understanding.

Hope not to bombard with too much info… but this summary high level article is also a good starting point to get familiar with the high level concepts around money transactions.

http://martinfowler.com/eaaDev/AccountingNarrative.html

Cheers…

Hi,

that’s a lot of info. But at the core of it, I see some need for being able to expose the aggregate state in some monitoring interface (not the normal, operational interface, for obvious CQRS reasons).

I always see 2 levels of support from the Axon perspective. The first is to make sure it’s possible, most likely requiring some custom coding. With the current state of Axon 2.0, that’s definitely the case. The second is to provide explicit components. With the current state of Axon, that’s not the case. Although there is work being done on the Event Viewer, which would be a good platform to incorporate this.

Once I have more time on my hands, I will read through the articles. Thanks for thinking along, though.

Cheers,

Allard