Axon Framework + Apache Druid for real-time analytics

Hi all,

I’ve been using Axon for a while now, and I’m thinking about supporting BI real-time uses cases by leveraging Apache Druid on the read model side.

Natively, Apache Druid supports Kafka, Kinesis, and CSV files as a streaming source.

I’m currently using Axon 4 with the Axon Server. I know I could use the axon-kafka extension so that Druid could ingest the data from there but I m not a big fan throwing in another piece of infrastructure knowing that Axon Server already offers streaming capabilities.

So my first idea is to write a Apache Druid extension that would make the bridge between a tracking processor for example and the Druid ingestion engine : https://druid.apache.org/docs/latest/development/modules.html#adding-support-for-a-new-input-source

Are you aware of any existing community project about this ? I’ve limited resources to work on it right now, but I guess this could benefit a lot of people wanting to build analytics on top of their business data.

Thanks,

  • Sorry, Apache Druid actually natively support Kafka, Kinesis and Tranquility and not raw CSV files as a stream source.

Hi Jerome,

I haven’t heard of anyone working on a Druid connector for Axon yet. Sounds like it shouldn’t be that hard. I’m wondering if you should even use the Tracking Processor at all. You could also use the connector directly (in Axon 4.4, we have separated the AxonServer Connector from the framework) and just open a stream to consume messages. I assume druid has a place to store the progress of the consumer, as they do mention that they use Kafka’s exactly-once semantics. Axon supports those same semantics.