How are people accomplishing BI from their Axon system? Event publishers, custom read views, direct event log access?

Imported from Google Groups
1

For all the great benefits of building an event sourced system one of the largest was how this event log is immutable and a permanent record that you could build some really good BI (Business Intelligence) data from. We’re now at the point where we need to start doing this and I’m curious how others have accomplished this without impacting the core system itself?

Our system is Axon 3.x at this time and events are backed in a SQL store (for now) and run in AWS. The hope here is that we can setup some data pipeline to offload the work away from our development team (or as much as possible) and provide our technical analysts to build their data set from various data sources.

Now our system has read views that we’ve built from our event stream already and we can point tools directly at this but I don’t like how they may impact a production system. There are also many un-realized read views that the system doesn’t need to operate on but BI will love to have. I was thinking it would be better to integrate an EventPublisher that would push the event stream out of our system and into something like SQS that kicks off the data pipeline. The downfall with that is possibly duplicate work/logic in a data pipeline that we already do to create an existing read view within the system.

I know this isn’t directly about Axon itself but I am curious how you have or how you would imagine accomplishing this within an Axon system. Maybe the answer is just “yes” and both direct DB read view access to existing plus an event publisher for custom is needed but I feel like I might also be missing something more obvious that provides a better balance.

Appreciate any positive or negative feedback from everyone’s varied experiences.

2

Hi Jeff,

these are indeed interesting topics. Actually, one of our customers that do this will be presenting at the Event-Driven Microservices conference at the end of this month (see https://axoniq.io/conference), and our partner Trifork will be giving a workshop on using the Event Store for Machine Learning and advanced analytics. If you have a chance to attend, don’t miss it.

What our customer did, in short, is basically build a separate set of models that contain the information the way the BI tools would like to use it. Sometimes they are similar to the view models, but at least any operations by BI tools will not have a performance impact on the operational systems.

Hope this helps.
Cheers,