For all the great benefits of building an event sourced system one of the largest was how this event log is immutable and a permanent record that you could build some really good BI (Business Intelligence) data from. We’re now at the point where we need to start doing this and I’m curious how others have accomplished this without impacting the core system itself?
Our system is Axon 3.x at this time and events are backed in a SQL store (for now) and run in AWS. The hope here is that we can setup some data pipeline to offload the work away from our development team (or as much as possible) and provide our technical analysts to build their data set from various data sources.
Now our system has read views that we’ve built from our event stream already and we can point tools directly at this but I don’t like how they may impact a production system. There are also many un-realized read views that the system doesn’t need to operate on but BI will love to have. I was thinking it would be better to integrate an EventPublisher that would push the event stream out of our system and into something like SQS that kicks off the data pipeline. The downfall with that is possibly duplicate work/logic in a data pipeline that we already do to create an existing read view within the system.
I know this isn’t directly about Axon itself but I am curious how you have or how you would imagine accomplishing this within an Axon system. Maybe the answer is just “yes” and both direct DB read view access to existing plus an event publisher for custom is needed but I feel like I might also be missing something more obvious that provides a better balance.
Appreciate any positive or negative feedback from everyone’s varied experiences.