Missing axon-message-timestamp from kafka headers

When playing back messages in our topic via kafkacat we are noticing missing timestamp values in the header. While stepping through the axon extension for Kafka we were able to confirm that the value is being populated after printing out the ProducerRecord. Can someone explain why kafkacat (or another suggested tool) is not displaying the value for axon-message-timestamp?

Producer Record before being sent to Kafka appears to contain an epoch time for the axon-message-timestamp

`
ProducerRecord(topic=enable, partition=null, headers=RecordHeaders(headers = [RecordHeader(key = axon-metadata-traceId, value = [53, 98, 99, 100, 48, 55, 56, 50, 45, 99, 50, 102, 100, 45, 52, 51, 48, 50, 45, 56, 53, 54, 54, 45, 50, 52, 48, 53, 53, 100, 99, 49, 52, 97, 97, 57]), RecordHeader(key = axon-metadata-correlationId, value = [53, 98, 99, 100, 48, 55, 56, 50, 45, 99, 50, 102, 100, 45, 52, 51, 48, 50, 45, 56, 53, 54, 54, 45, 50, 52, 48, 53, 53, 100, 99, 49, 52, 97, 97, 57]), RecordHeader(key = axon-message-id, value = [56, 51, 48, 98, 100, 54, 52, 99, 45, 100, 51, 50, 101, 45, 52, 98, 99, 53, 45, 98, 53, 50, 50, 45, 100, 54, 50, 98, 97, 48, 97, 99, 53, 57, 48, 50]), RecordHeader(key = axon-message-revision, value = [49, 46, 48]), RecordHeader(key = axon-message-timestamp, value = [0, 0, 1, 109, 23, -96, 12, -124]), RecordHeader(key = axon-message-type, value = [99, 111, 109, 46, 97, 114, 97, 109, 97, 114, 107, 46, 101, 115, 115, 46, 99, 111, 109, 109, 111, 110, 46, 83, 121, 110, 99, 72, 111, 115, 116, 67, 114, 101, 97, 116, 101, 100])], isReadOnly = false), key=null, value=[B@6f0d9ece, timestamp=null)

`

When viewing the record through the lens of kafkacat the header appears to contain all the values with the exception of axon-message-timestamp

`
Headers: axon-metadata-traceId=5bcd0782-c2fd-4302-8566-24055dc14aa9,axon-metadata-correlationId=5bcd0782-c2fd-4302-8566-24055dc14aa9,axon-message-id=830bd64c-d32e-4bc5-b522-d62ba0ac5902,axon-message-revision=1.0,axon-message-timestamp=,axon-message-type=com.aramark.ess.common.SyncHostCreated

`

Hi Jerry,

I’ve heard this issue recently and to be honest, I am not settled yet why this happens.

If you check the Kafka Extension, more specifically the DefaultKafkaMessageConverter#createKafkaMessage(EventMessage<?>, String) message, you can see that it’ll create the ProducerRecord providing null for the timestamp.

If you then check the class javadoc of the ProducerRecord, you can see the following:

The record also has an associated timestamp. If the user did not provide a timestamp, the producer will stamp the record with its current time. The timestamp eventually used by Kafka depends on the timestamp type configured for the topic.

  • If the topic is configured to use CreateTime, the timestamp in the producer record will be used by the broker.
  • If the topic is configured to use LogAppendTime, the timestamp in the producer record will be overwritten by the broker with the broker local time when it appends the message to its log.

In either of the cases above, the timestamp that has actually been used will be returned to user in RecordMetadata.

This would thus lead me to ask how you’ve configured the topic, as that might impose the issue you’re seeing.
Maybe the kafkacat tool you use is inserted right before the Produce is allowed to set the timestamp?

That’s however something I am not completely sure of, as I have no direct experience using said tool.

Hope we can figure this out!

Cheers,
Steven

Thanks for the information.
We’l digest and investigate further

Jerry

Hi Jerry,

I’ve heard this issue recently and to be honest, I am not settled yet why this happens.

If you check the Kafka Extension, more specifically the DefaultKafkaMessageConverter#createKafkaMessage(EventMessage<?>, String) message, you can see that it’ll create the ProducerRecord providing null for the timestamp.

If you then check the class javadoc of the ProducerRecord, you can see the following:

The record also has an associated timestamp. If the user did not provide a timestamp, the producer will stamp the record with its current time. The timestamp eventually used by Kafka depends on the timestamp type configured for the topic.

  • If the topic is configured to use CreateTime, the timestamp in the producer record will be used by the broker.
  • If the topic is configured to use LogAppendTime, the timestamp in the producer record will be overwritten by the broker with the broker local time when it appends the message to its log.

In either of the cases above, the timestamp that has actually been used will be returned to user in RecordMetadata.

This would thus lead me to ask how you’ve configured the topic, as that might impose the issue you’re seeing.
Maybe the kafkacat tool you use is inserted right before the Produce is allowed to set the timestamp?

That’s however something I am not completely sure of, as I have no direct experience using said tool.

Hope we can figure this out!

Cheers,
Steven