avro serializer

We are considering using avro for event serialization. It is mentioned in the docs sometimes as a valid (and possible better suited than plain json) … but it comes with the additional complexity of dealing with schema resolution and schema versioning …

Before we start digging deeper and implementing, I wondered if there are any sources or success stories in the community already that we could build on and/or reuse …

Thanks you
Jan

Hi Jan,

I have not been involved in using Avro for event serialization, but I’m very interested if the community had some experience with it so far.

I have couple of criteria for choosing correct serializer (Avro/LinkedIn, ProtoBuf3/Google, Thrift/Facebook, Wire/Square, Thrifty/Microsoft, … )

  • I like to think of the schema of our messages as a contract (or at least a big part of it). How this transfer protocol supports Schema Evolution / backward compatibility / forward compatibility?
  • Kotlin support?
  • Speed/latency?
  • Who is behind it? Which company/community?
  • What tools we have for code generation?
    ProtoBuf3 and Wire are on top of my list :slight_smile:

Best,
Ivan

Hi Ivan,

Very good idea to do it more formal. I fully agree with the format idea.

I consider the following additional criteria as important:

  • Schema validation
  • Cross platform support

Why are you connecting Avro to LinkedIn (which is facebook in the end)? To me, Avro is Apache Avro and is kind of independend indistry standard (https://avro.apache.org/)
Looking on the contributor list: https://avro.apache.org/credits.html I can’t clearly identify the “company” behind it…

Cheers,

Simon

Hey Simon,

Yes, you are right regarding Avro - LinkedIn. My mistake, thanks!

  • Schema validation is the first criteria (somehow, I consider it under the Schema Evolution).
  • Cross-platform support - Yes, it is important! My focus was limited to JVM (Kotlin, Java) in this case.

I can add one more: Are there any frameworks like PACT or Spring Cloud Contract that can support us in writing Consumer/Producer Driven Contract tests for the chosen serialization protocol.

This can be important in a distributed environments (microservices) where your contract/api should be a schema. Please, mind that validating the schema in event driven systems (with event sourcing) can be a challenge, as we have old events still in the system, and we have to pay attention to backward compatibility and versioning in this case. Some protocols like Protobuf have this built in within, some don’t, and we should consider this.

Please, continue sharing your ideas and remarks on this topic. It is indeed the interesting one.

Best,
Ivan