Hey everyone, my two cents.
First, @Steven_van_Beelen I can appreciate that this is not the only thing on your plate, but this is a huge bug that affects anyone who’s running Axon + Hibernate in a multi-tenant environment, which is bound to be a significant number of people. I honestly can’t think of a more fundamental problem than “silently and randomly not running event handlers”. I understand that you’re going to solve this in Axon 5, but this thread is over 2 months old, and (unless I’m missing something), there’s not a word about this in the docs. This should be plastered in big red letters at the top of the documentation, and it takes 5 seconds to do, saving others man-days, even man-weeks, trying to diagnose the problem. We were lucky to encounter this topic, others might not have been. I don’t mean to be disrespectful, but this an absolute no-brainer and should’ve been done immediately when you found out this was an issue. Please prioritize it, and let’s get it into the docs in big red letters ASAP.
Moving on, the workarounds described above do not actually fix the issue - they just make the window for the race condition smaller. Setting the allocation size to 1 just causes Hibernate to query the database each time before it attempts to save a record, as can be seen on the attached screenshot. Apart from this doubling the load on the DB, imagine Tenant A and Tenant B - Tenant A asks the DB for an ID, gets 40, but before it manages to persist the event, Tenant B also asks for an ID, gets 41, and manages to persist its event before Tenant A does, leading to out of order globalIndex.
The way to solve this is to switch the generation strategy to identity, and migrating the columns to BIGINT GENERATED ALWAYS AS IDENTITY. This can be done using an AdditionalMappingContributor, which I’m attaching bellow. Note that you also need to migrate the columns in the database as well.
@AutoService(AdditionalMappingContributor.class)
public class AxonModelReplaceGeneratedValueIdsWithAutoIncrementAdditionalMappingContributor implements AdditionalMappingContributor
{
@Override
public String getContributorName()
{
return AxonModelReplaceGeneratedValueIdsWithAutoIncrementAdditionalMappingContributor.class.getSimpleName();
}
@Override
public void contribute(
final AdditionalMappingContributions contributions,
final InFlightMetadataCollector metadata,
final ResourceStreamLocator resourceStreamLocator,
final MetadataBuildingContext buildingContext
)
{
for (PersistentClass entityBinding : metadata.getEntityBindings()) {
if (AbstractSequencedDomainEventEntry.class.isAssignableFrom(entityBinding.getMappedClass())) {
remapToAutoincrement(entityBinding, "globalIndex");
}
if (AssociationValueEntry.class.isAssignableFrom(entityBinding.getMappedClass())) {
remapToAutoincrement(entityBinding, "id");
}
}
}
private static void remapToAutoincrement(
final PersistentClass entityBinding,
final String fieldName
)
{
var value = (SimpleValue) entityBinding.getProperty(fieldName).getValue();
value.setIdentifierGeneratorStrategy("identity");
Column column = CollectionUtils.extractSingleton(value.getColumns());
column.setSqlType("bigint");
}
}
Example Postgres migration (be sure to review and edit/substitute to match your environment):
ALTER TABLE domain_event_entry DROP CONSTRAINT domain_event_entry_pkey;
ALTER TABLE domain_event_entry
ALTER COLUMN global_index DROP DEFAULT,
ALTER COLUMN global_index SET DATA TYPE BIGINT,
ALTER COLUMN global_index ADD GENERATED ALWAYS AS IDENTITY;
ALTER TABLE domain_event_entry ADD PRIMARY KEY (global_index);
DROP SEQUENCE IF EXISTS domain_event_entry_SEQ;
ALTER TABLE association_value_entry DROP CONSTRAINT association_value_entry_pkey;
ALTER TABLE association_value_entry
ALTER COLUMN id DROP DEFAULT,
ALTER COLUMN id SET DATA TYPE BIGINT,
ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY;
ALTER TABLE association_value_entry ADD PRIMARY KEY (id);
DROP SEQUENCE IF EXISTS association_value_entry_SEQ;
