What is the current thinking of supporting a no-sql backbone for all dimensions of Axon, eventStore, Saga persistance, Scheduleing event persistance, query db repository with a distributed nosql stack.
Seems like one platform end to end would really enable multinode footprint for no sql infrastructure and support availability. Also, load balancing multiple jvms with disteibuted bus and message queuing persitance for multi-node bus is a ?
Could Mongo provide full end to end capability where no sql jpa infrastructure is needed. What would be the alternative to Quartz? especially in multi-node environment?
Mongo support is going to be supported (meaning not in incubator anymore) from Axon 2. In fact, yesterday I’ve spent some time on building a Serializer that serializes any object to a Mongo BSON model, so that they can be stored as native Mongo documents (in fact as part of a document). Meanwhile, Jettro is busy implementing the SagaRepository using Mongo. It’s already in the incubator of 1.3. For an example, check out the Axon Trader sample application: https://github.com/AxonFramework/Axon-trader
As for scalability related command and event busses, some will be included in 2.0 as well. I’ve already built the dynamically scalable distributed commandbus, that does automatic routing of commands for optimal performance. We’ll also soon start with an AMQP-based event bus soon. It will allow you to share events among different nodes using an AMQP compatible Message Broker, like RabbitMQ.
An alternative for Quartz wouldn’t really be needed. Quartz has pluggable storage, so you could use MongoDB to store the jobs for quartz. An Quartz is also capable of working in multi-node environments.
I can’t really give a timeline for all this yet. It depends a bit on the amount of time I can spend on it. I’m working on a sponsoring deal that will allow me to speed up things a bit. I’m hoping to be ready this summer.
Thanks Allard… lot of work in progress and exciting stuff…
One more question is how to manage Axon scale with traditional java web app jvm hosting webservices/api end points or web app site.
Together or separate configuration options. Elaborating on those design techniques for scalability also is a interesting area.
e.g. traditional webapp compile could run axon within jvm, then how to scale horizontally with that topology.
Alternatively webapp scaling could be seperated from axon i.e. separate jvms and remote connectivity into different types of clusters one for webapps and one for the axon jvms and another one for persistance multinode backbone.
These are some important design paradigms to consider and solve for dimensions. Any best practice for these too would be key for broader developer education and adoption and any framework simplification would also help adoption curve.
Thanks again for all the exciting work in this area.
I am curious to know your motivation for nosql event-stores.
To me a file is the perfect store for ARs.
The notion of a FileStream, the way you read it, the way you append
events, fits perfectly.
There are small concerns though about integrity across AR's. But the
problem did not arise to me yet.
I've started to use CQRS, thanks to axon, on two projects, until now I
really don't see the benefit of (no)sqldata-bases on the command side.
On the read-side I use a nice, productive, mongodb.
And there is really no need for a CQRS-framework that implements read-
model stuff.
This is a philosophical question… I will try to be breif
My 2 cents… The main motivation is for horizontal scale, redudancy and availaibility of the AR/SAGA/Scheduled Event data along with read only… and drive for consistency for persistance across cqrs stores and read only stores to simplify persistance management for all data in a multi node infrastructure… and simplify maintenance and management for availability/redundancy and scalability
the file system event store may be a fast and simple one, but it doesn’t provide any failover capabilities. It simply requires you to always send commands for one aggregate to the machine that hosts the event store for that partition. You could share the event store using an NFS, but you’re probably shooting yourself in the foot there.
Nosql databases acknowledge the need for certain scalability aspects (see CAP theorem) very well. Mongo is one that fits the typical Event Store requirements (CA, no P). Having them will allow you to scale dynamically as well as provide an easy infrastructure for failover.
Thanks, I understand the CA(P) idea. Still I suspect there is a
simpler solution to reach consistency and availability than using a
database that is build for a different use-case. We have done this for
decades with objects in SQL, and now we should not repeat the same
mistake with Events in NoSQL. As a business leaning dev I am weak in
the technical details but consistency and availability of a file can't
be a harder problem than with a document database. Given the
prominence of files in unix systems I can not get rid of the
impression: this is a solved problem. Again as a layman: SAN or NAS
e.g. provide automatic replication.
This is just to evolve the idea, getting a clearer picture...
document databases like MongoDB are a much better match for storing Object than relational databases. Other than that, I don’t really see a problem in using databases for storing application state. The problem is in the devs, that expect not to have to think about storage at all. ORM is a craft, not a tool.
It might be possible to use a NAS and a file-based event store to achieve scalability. However, I seriously doubt that it will be any easier than using a Database. Mongo solves much of the scalability, replication and sharding issues that you need to do yourself when using a NAS. I wouldn’t go there…
I know its not quite summer yet… Any pointers on how to pilot multjvm webapp with embedded axon to work in distributed node environment? Everything works great in 1 JVM, need to learn to start figuring out multinode…
Sample use-case…
need single webapp like axon-trader on two tomcat or any other webserver jvm load balanced nodes
traffic hitting each webapp should work for full command bus and related autonomity on the same jvm to manage transactions and complete all event handlers before commiting (axon magic)
removal of one node from the webapp cluster should keep app running smoothly
scale to 3 or 4 nodes.
Now behind the scenes multinode cluster for mongo or mysql, AMQP cluster for message processing and pegging command bus for full eventhandler chain on one aggregate in one machine, while queuing subsequent incoming domain commands to same aggregate and keeping update and eventhandler integrity for the same aggregate, multinode redundancy at command bus level. Key here is blocking the second thread from returning untill all eventhandlers of 1st request from 1s user and 2nd request from 2nd user are done (this is key Axon feature and not conceivable in pure async event handler design).
Any ideas or pointers… could not quite figure it out in axon-trader… seems like things have been quiet for couple of months.
Reviewed disruptor and jgroups base distributed bus code… interesting stuff there… any priority on Spring Integrtion with rabbitmq multinode clustering based distributed bus or am I missing something in connecting the dots… thanks and Cheers
Axon trader is still an single-JVM solution, but we will start migrating
that to Axon 2.0 shortly. We can only start that when the Axon 2.0 API is
completely final. The idea is to base the migration guide on the findings
we have there.
The distributed command bus is as good as finished. It can use Jgroups to
dynamically connect and distribute command across members. That means that
when a member drops, its share of the load is distributed over the
remaining members.
On the Event Bus side, I am working on AMQP connectors. At a client, where
I am helping out with scaling their application, I noticed that using
Spring AMQP provides a lot of features that deal with resilience, such as
automatically reconnecting when a connection is lost. So the first solution
will probably build on that. You can always use the Spring Integration
connectors (that have been around since the early Axon days) to connect
with other protocols.
The use case you describe can be easily achieved by setting up a
distributed command bus on each Tomcat node, with a SimpleEventBus to
dispatch events locally. The distributed Command Bus will ensure that
commands are consistently routed across available members. The
SimpleEventBus will ensure that Event Handling (and thus query database
update) is done in consistently and in the same transaction. You can use a
single (clustered) database behind the machines.
Note that the consistency requirement puts the bottleneck in the database.
If you don't need your QueryDB to be absolutely consistent, you can use an
asynchronous events bus, such as the ClusteringEventBus with an AMQP
connector.
Thanks for the quick reply… this definately helps… quick question i had was
that. instead of using jgroups for reselience on command bus distributed design, would Spring AMQP provide the same capability? Couldn’t you use the same stack where distribution logic of the command scaling to multiple nodes for the command handling with local event bus for transaction integritiy or distributed event bus cluster (async) can be managed by same featues of AMQP infrastructure? Could that help simplify the infrastructure setup and maintenance of these cluster types?
Again, thanks for the brillliant work on this framework… and looking forward to multinoding…
I’ve had thoughts in the same line. The issue I believe is that there is some more coordination to be done with the commands, making sure that commands to the same AR always ends up on the same node as long as its alive and keeping track of who’s alive and which ARs they are responsible right? If the command nodes are not fully in sync as to what state they have in memory that is…
What about using an MQ for the command messages and groups just for the coordination?
Initial thoughts is to really figure out a way to make the infrastructure experience for scaling/resilience and monitoring all of this in AMQP infra and eliminate the jgroups maintenance and management/complexity variable, however the design challenge is to figure out how. Can a dynamic pub/sub and dynamic topic subscription type 1 to1 can solve in some combination for coordination challenge (query is AR active in a multinode queue)… something like the quartz cluster design where it does not process the same event twice if it is being processed on one node is the kind coordination solution to strive for… maybe jgroups is not a big deal?
Another thought is could a persistant cache for active ARs(Where they are and what they are doing) be done in mongo/mysql or some shared cache and a isActive check done before starting new operation and once found route it to the listener queue for that node for incoming ARs (alos these queues could be multithreaded to boost async performance). This query check has performance implications, however don’t know how jGroup coordination can avoid that as synching consistency is also a challenge unless not guaranteed with transaction in persistent area, for resilience and preferably in a cache to speed things up. if mongo multi nodes were on same machine they would probably get data from that proximity for speed… ?
I understand where the idea comes from. Some colleagues have build a
solution using an AMQP broker with multiple nodes processing the commands.
Their conclusion: don't!
The reason is that there is a major design aspect about AMQP, that doesn't
make it suitable for consistent routing. The idea behind message brokers is
that it abstracts sender and receiver away from eachother. The sender just
sends and "assumes" that any client will receive the message. The problem
is that AMQP will just place it in a queue.You cannot dispatch consistently
from a queue to its recipients. If you don't care which nodes handles which
command, a single queue for them to read from is your solution. If you want
to consistently route commands, AMQP is not the design you're looking for.
"and eliminate the jgroups maintenance and management/complexity". What's
so complex about JGroups? All you need is to configure a list of IP
addresses or a gossip router and you're set. It's really simple. And the
biggest advantage is that Axon will use JGroups to dynamically (and
consistently) route commands to the members. This will ensure that commands
for the same aggregate will always hit the same server. So it's perfectly
safe to use a cache. The power in this solution comes from the lack of the
need for a centralized registry of who does what. Such a registry would
only be a single point of failure, or at least a severe bottleneck.
"What about using an MQ for the command messages and groups just for the
coordination?"
I don't see how this makes the world any easier. JGroups is perfectly fine
for point to point communication. Adding MQ to the mix will take everything
to another level of complexity where, honestly, I don't think the tools
will really benefit from eachother.
Agree… Feels like if the jgroups behind the curtain implementation management is hidden away with just few config ip/addresses, then it should be ok, however the resiliance and scale of that will be key (bringing down or adding nodes on the fly and down node detection to facilitate bringing up new nodes without compromising date integrity and state).
JGroups comes with a pretty good default configuration for TCP, UDP and the
like. All you need to configure is IP addresses. In Axon, I will provide
easy access to some of the sensible default settings. Of course, advanced
tuning will always require a bit more jgroups knowledge.