Approach for dealing with large binary data in ocassionally connected systems

Hi,

we plan an application with ocassionally connected clients. The plan is to run the same (event-sourced) model on clients and server and (while offline) store all commands on the client (plus apply them locally) and apply them later to the server.

Some commands contains large binary data (namely pictures). One constraint is that it should be possible to synchronize all outstanding commands independent of the binary objects (to allow fast mobile sync and sync the binaries later).

So my idea is to use the following approach:

On clients and server exists a “binary repository” that simply stores binary objects (giving them a unique id).

  1. When a picture is added
  • the picture is put to the reposititory
  • a “AddPicture” command is issued, containing the uid of the picture
  • the model will then fire a “PictureAdded” event.
  • if the command fails, the picture is removed from the repository

When synchronizing the commands with the server, the client would simply send the “AddPicture” command to the server (without binary data). In this case the “PictureAdded” event would be created on the server althought the picture is not yet in the repository. The read side would detect this when someone want to see the picture and display some “not yet synchronized” message.

When synchronizing the binary data, a client would simply put all outstanding binary objects to the server’s repositoryand receive all new binaries from the server.

After synchronisation (commands from client->server), the client would load the whole event database from the server and to a local replay.

  1. When a picture is deleted
  • a “RemovePicture” command is issued, containing the uid of the picture

  • the model will then fire a “PictureRemoved” event.

  • an event listener to this event would register a post-commit hook and then the event is committed, the picture would be removed from the local repository

When synchronizing the commands with the server, the server would also create a “PictureRemoved” event and the picture gets removed from the server’s repository. After synchronizing to other clients, the picture would also get removed from them.

What do you think about this approach? Could I do it in a better way? I thought about embedding the picture data in commands/events but I guess that this is a bad idea. And I could not separate data synchronizing from picture synchronizing.

Comments are welcome :wink:

Kind regards,
Michael.

Hi Michael,

this is exactly the way I deal with large binary data. One minor difference is that I like to use content based identifiers. Typically, this would be a hash of the contents (e.g. SHA-1). This way, it’s easy to synchronize binary content between client and server.
In some applications, we chose not to delete binary data, since it is referenced from the events in the event store. The binary data is seen as part of the audit trail.

Cheers,

Allard