Microservices Ecosystem Design Question

[CatalinCaprita]

Greetings to everyone and let me start by saying how much I appreciate the dedication and community-centered approach of this framework’s team! Kudos to you guys!

A little background: I have been responsible for researching and deploying microservice-migration solutions for our development team, as the next milestone for us is to migrate existing monolithic suite of applications to an new microservice ecosystem. I have been familiarizing myself with the Axon Framework, Spring Cloud extension and Spring AMQP extension for the past few months and it has been a successful discovery in our team. It has been agreed that Axon will be the main framework for the above mentioned migration process!

After migrating one of the monoliths successfully, I’ve come across a different challenge that I would love to receive input on. The microservice system will have one main ‘central’ application in that this one service will be responsible for integrating with an ERP and act as our own data warehouse. This will replace the current approach in which each application directly integrates with the ERP. As an effect of the periodic data synchronization between Central-App and the ERP, all microservices should be informed to start a synchronization of data received by the Central App.

1. The first thought I had in mind was to send an individual command from within a Saga, for each Microservice, during the syncing, i.e. App1SyncClientCommand, App2SyncClientCommand, and so on. The main problem here I believe would be the amount of overwhelming amount of commands as there will be one for each client for each application.

2. The second approach i have thought of would be a PULL approach, that is after a Client’s data has been synchronized centrally, an Event will be dispatched to all microservices, making them aware that there is new data to be synced locally. In turn, each microservice could send some sort of PULL command for an Aggregate, or even an Event that would let the Saga know, there is a new Subscriber that needs the newly synced data.

That would be my main concern, but I would also be curious about another design aspect: The ClientAggregate is present inside each of the microservices. Is there possible, and if so would you encourage, having this common Aggregate inside something like a shared-module to be visible for each microservice or will it be better to have a separate implementation?
Thanks for your attention!

[Bert_Laverman]

Catalin,
thank you for your question and welcome to the world of Axon!

I think, from reading your story, that what you are missing can be found in Domain-Driven Design. The way you describe your proposed solutions, you are viewing each microservice as a separate domain, each with its own commands, queries, and events, whereas it would help to view the domain functionally. You haven’t told much about the domain of your “Central-App”, but it is fed from an ERP system. If the application (as in referring to the collection of microservices you describe) also belongs to the same functional domain as the ERP system, then every (synchronization) event from the ERP system is valid for the app and can practically be passed on as is. If the domains are different, you would typically use an anti-corruption layer, filtering and translating the ERP events into events relevant to the app’s domain. You might even have larger plans for the future, that would make the layer part of the ERP domain, translating everything into more globally relevant events.

The app then can use Sagas as you describe, but not generating microservice-specific commands and events, but rather local-domain specific ones. Using a distributed bus you can then allow the microservices to handle these. The data-warehouse you describe would be your eventstore, which collects and records all events, allowing replays for view-building. You’ll forgive me for suggesting to take a look at Axon Server, which can help you with a consolidated component to provide infrastructural support for this. The Open Source Axon Server Standard Edition will help you get up and running quickly in a single-domain scenario, with Enterprise Edition adding multi-context (domain) support, more advanced security, and distributed deployment options.

For the Aggregates a similar story can be told, as they appear to be common across the domain and then can be shared as you suggest. If they are set up as Event Sourced, you would not need all microservices to do their own pull-style synchronization, but rather have that done once. Any microservice needing information from these would gather this using queries on the view models that have been built using the same events used to store the state of the aggregates, again relieving them from keeping their own copies of the aggregates. These view models can be optimized for the types of queries you would place on them, and either persisted in a model-fitting solution or kept in memory and rebuilt on need. This will depend on the size of the model, and the options you have for keeping in-memory models alive long enough for the expected use.

So, all in all, a lot of choices available to grow this solution further. Feel free to contact us if you want to take a more in-depth look at Axon Server.

I hope this helps you along!
Cheers, Bert Laverman

[milendyankov]

I moved the topic to #architectural-concepts as it’s not really about Axon Framework

[CatalinCaprita]

Hello Bert,
Thank you for such a comprehensive answer at such short notice! I took the time to forward your response to the team, and actually tried to come up with a diagram that would better describe our situation.

This part of your answer would best describe what our current monolithic architecutre is doing, and what we will be using in this new one as well, since the ERP system is more of a third-party system rather than our own.

@Bert_Laverman

If the domains are different, you would typically use an anti-corruption layer, filtering and translating the ERP events into events relevant to the app’s domain.

To elaborate on the diagram. Currently this A entity is the Client. The Central App currently runs a synchronization process during which new data is pulled from the ERP and persisted inside our own system. This Client Aggregate contains data that are manipulated and used throughout our whole system by all the microservices. Whenever The Central App receives updates about the Clients, it will notify the microservices to individually record these changes. Upon reflecting on this scenario, your suggestion,

@Bert_Laverman

The data-warehouse you describe would be your eventstore, which collects and records all events, allowing replays for view-building.

Seems to be the answer we were looking for. However, I would like to know if I understood it correctly:
You are suggesting that we should maintain this Aggregate only in the Central App and make all the other Microservices aware of the Central App Event Store. This way, whenever the synchronization for a Client succeeds, by using the Distributed Command Bus (which I forgot to mention we are already using) we could send simpler Commands, containing something like the client’s ID for every microservice. Each MS will in turn perform EventSourcing for that Client’s ID directly from the Central EventStore, thus skipping the overhead of a separate implementation of the same aggregate?

Additionally, I would like to point out that different microservices operate on different models from our domain, hence our implementation of different aggreagets and event stores inside each microservice (as is the case with B,C and D in the diagram. These can be created, updated and deleted from specific microservices only, but the changes should be reflected in the central app as well. Would you advise against this in favour of directly forwarding the commands to the central app?

But won’t such an approach be a little too much to handle for the central EventStore in terms of performance with so many transactions going on, thus impacting the overall availability?

I am looking forward for your answer and kind regards!

[Simon_Zambrovski]

Hi Catalin,

Very interesting and very fundamental question in the same time…

I think there are several ways to address and answer your question. And I assume that currently you are looking for the technical integration solution and Bert is pulling you to the strategic design of DDD.

Let me try to help here and put my 5 cents in…

Generally speaking there are three types of messages you can use: Command, Event, Query.
Using command means that your central app pushes changes to the microservices. Using events means that you just use event-streaming by distributing your event bus and the central app is broadcasting the change. Using queries - the single components can subscribe for changes and the central app might implement query subscription handlers pushing the changes. All these solution will technically work (some of them only with and others also without Axon Server).

Now let return back to Bert’s important questions. The distribution your drawn on the diagram is about technical separation of the system components, not about logical. In terms of DDD the strategic design phase is all about modelling your core domain and identification of the bounded contexts and their relationships. The bounded context is your logical boundary - and it is mostly that a technical boundary of deployment is there in case you distribute… you can distribute your bounded context over several microservices even further e.g. to meet some non-functional requirements. The reason why it is so important to understand if the components belong to the same context or not is the question of relationship between them.

If they belong to the same bounded context, they share the same language (domain types, commands, events and queries) and might by synchronized via event stream without any problems.

If they don’t belong to the same bounded context, it is of utmost importance to understand and classify the relationship between them, grade of freedom, independence of modification, life cycle and coupling… In the worst case you will NOT be able to expose your “internal” event stream to the components and have to adapt your means to the official API (commands and queries).

I hope you understand why you need to start from modeling and strategic design… To get further, I would like to understand, WHY you cut the microservices out of your central app the way you did?
What do you expect them to do better as being separated than being a monolith?

If it is like I expect, the event stream from central app is not going to work. You will need to declare a public API of every microservice in terms of commands and queries and solve a multi-context communication. For doing so, there are two approaches: orchestration and choreography.

For an orchestration aspect, you have a special “process” / “orchestrator” entity that is responsible for the communication between the contexts. In doing so, it will be in charge of doing all the translation between the commands and queries of different bounded contexts.

For choreography, there is no central process knowledge, but you translate your internal domain events into “milestone” / “pivot” events and implement components reacting on them.

Your next question is what to use… It depends, but the general rule is: if the communication can fail (not technically, but because of illegal data or state of the application), you will need to be able to react on that (and compensate) and the orchestration is the way to go. If it is all about executing unconditional steps one after another without any compensation, the choreography will work…

I hope I could help a little.

Cheers,

Simon

[CatalinCaprita]

Greetings, Simon!
First of all I would like to assure you that your 5 cents have not been wasted, as your response generated several rounds of discussions in our team! After reading your response, we took a step back and started reviewing the logical design of our Domain.

I would like to clarify some aspects that seemed to pose some challenges to you on coming up with an answer. Do keep in mind that we are still under discussions, thus they may not be as comprehensive as expected.

@Simon_Zambrovski

To get further, I would like to understand, WHY you cut the microservices out of your central app the way you did?
What do you expect them to do better as being separated than being a monolith?

We decided that CentralApp 's main goal to aggregate data about our Clients i.e. Entity A, so A will be present only in the CentralApp. Our main purpose for doing this is that until now each monolith stored separate information about the same Client.

In the old architecture, if an admin wished to retrieve all data about a Client, he would have had to browse each application and manually put the data together. We decided it would be better to have them all persisted in one place, mainly to facilitate this sort of reporting scenario. As a result, A will only appear in CentralApp. Now, say Microservice2 is responsible for managing all sorts of Invoices (Entity B), Letters(Entity C). Additionally, say Microservice3 will be concerned with Offers(Entity D) and Offer creation flow. I can see from your suggestions and answer that 3 and 2 will not share bounded contexts. However, Offers are made to Clients, Invoices and Letters are sent to Clients, thus CentralApp needs to be aware of the changes made in each microservice. So, from a logical point of view, Microservice 2 needs to ditribute Invoice bounded context with CentralApp, and the same goes for the other micrservices. Do note that these microservices will also manage aspects that are not related to the Client at all, thus the separation.

1. Following both your and Bert’s recommendations we thought that event streaming will be a viable solution, i.e. whenever a client-related Model is updated, it will notify both the CentralApp and the respective Microservice. However, this would imply that, in a CQRS architecture, both Query sides will persist some of the same data. Do you think it is still advisable to follow this approach?
2. A second approach we thought of was to delegate a Saga to orchestrate this data-gathering process for each client, thus managing all the separate bounded contexts involved. However, the problem here is that this reporting feature is not automatic, but rather initiated by a user, so it will take an incredibly long time to manage the reporting, leading to low availability.

I hope I was able to clarify the aspects that have resulted in your confusion and would love your input on our ideas!