kafka and upcasting

Imported from Google Groups
#1

Hi,

I was exploring upcasting and I was wondering how to handle the case of events that have been published to Kafka that require upcasting. I was going through the code and it looks like the amqp and kafka modules do not support upcasting.

I can only think of two options so far:

  • Upcast the events as they come in from Kafka
  • Delete the topic and republish the upcasted event stream from the event store to kafka again

Both options would be problematic for existing consumers of the event log (Kafka).

Are these viable solutions? I’m hoping for more elegant alternatives.

Thank you,
Mik

#2

Any suggestions here?

#3

Hi Mik,

you are right in that the KafkaMessageSource doesn’t support upcasting at the moment. The Kafka connectors are a fairly new addition to the framework, and need a little time to mature.

The best way to implement upcasting, is to upcast messages as a consumer reads them from Kafka. Republishing to Kafka is problematic in most cases, and we generally advise against such strategy.
You can customize the KafkaMessageConverter to modify the structure of incoming messages, before passing them to Axon as messages. This will most likely be the approach we would take once we start implementing upcasting in the KafkaMessageSource.

Note that the primary usecase for upcasting is processing old events. If you are looking for a strategy to upgrade application versions and still be able to read messages produced by an older version, we recommend using forward and backward compatibility practices instead. Since Kafka may also be used to replay events since “day 1”, there could be a legitimate case for upcasters.

Hope this helps.
Cheers,

Allard