Spring boot app failed to receive Axon events in EventHandler methods

Hi all,

Apologized ahead because I am a total noob in Axon framework. Here is what I need to achieve… I have an Axon framework application that I need to connect to (its event bus) and receive events and process them.

If you need to know the version of various pieces, just ask because it is too much to post here.
When I startup my spring boot app I did not get any errors, here is a small part of the startup logs:

  restartedMain] com.feexx.reporting.config.AxonConfig   : Setting up TrackingEventProcessors for threads, batch size and other configurations... annotated with @ProcessingGroup...
  restartedMain] org.reflections.Reflections              : Reflections took 75 ms to scan 1 urls, producing 3 keys and 3 values
  restartedMain] com.feexx.reporting.config.AxonConfig   : Configuring EventHandlerProjector Bean 'packageRecordProjector' with maxThreads: 20 and batchSize: 10.
  restartedMain] com.feexx.reporting.config.AxonConfig   : .. 'packageRecordProjector' successfully configured with processing group 'Package-Record'.
  restartedMain] c.f.reporting.config.ApiSecurityConfig   : Entering configure
  restartedMain] o.s.s.web.DefaultSecurityFilterChain     : Will secure Ant [pattern='/**'] with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@75fa125e, org.springframework.security.web.context.SecurityContextPersistenceFilter@44a75ea3, org.springframework.security.web.header.HeaderWriterFilter@273df092, org.springframework.web.filter.CorsFilter@2c9dfe45, org.springframework.security.web.authentication.logout.LogoutFilter@52820678, com.feexx.reporting.config.ApiKeyAuthFilter@32299497, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@38bac51d, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@3df73d5e, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@25ea9ed1, org.springframework.security.web.session.SessionManagementFilter@22153964, org.springframework.security.web.access.ExceptionTranslationFilter@45b5769d, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@2d8cfe89]
  restartedMain] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
  restartedMain] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
ckage-Record]-0] o.a.e.TrackingEventProcessor             : Worker assigned to segment Segment[0/0] for processing
ckage-Record]-0] o.a.e.TrackingEventProcessor             : Dispatching new tracking segment worker: TrackingSegmentWorker{processor=Package-Record, segment=Segment[0/0]}
ckage-Record]-1] o.a.e.TrackingEventProcessor             : Fetched token: MergedTrackingToken{lowerSegmentToken=MergedTrackingToken{lowerSegmentToken=IndexTrackingToken{globalIndex=209300}, upperSegmentToken=IndexTrackingToken{globalIndex=209300}}, upperSegmentToken=IndexTrackingToken{globalIndex=209300}} for segment: Segment[0/0]
  restartedMain] i.a.a.c.impl.AxonServerManagedChannel    : Requesting connection details from axonserver:8124
  restartedMain] i.a.a.c.impl.AxonServerManagedChannel    : Successfully connected to axonserver:8124
  restartedMain] i.a.a.connector.impl.ControlChannelImpl  : Connected instruction stream for context 'default'. Sending client identification
  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8085 (http) with context path '/reporting-etl'
  restartedMain] c.f.reporting.ReportingEtlApplication    : Started ReportingEtlApplication in 11.918 seconds (JVM running for 12.809)

The spring boot app uses some custom code to hook into the Axon server. I will post it if anyone wants to see it.

So I went into the axon dashboard app and see our main axon app, the axon server, my spring boot app both connected. But when I generate an event from the axon app, I see the event dispatched into the handler methods of the axon app, but I did not see anything on the spring boot app as if no events was fired (in other words, the spring boot app did not see any events). I can also find the events in the axon dashboard (search) but I just don’t see any indications that it is dispatched and handled in the spring boot app.

Could you all give me some hints on how to perform diagnostics to check if the event bus and event handling methods in my spring boot app actually works? Any ideas is appreciated.

Regards,
Ed.

Hi Ed.

Can you share the content of axon-server log file please?

Arnaud

Hi Arnaud,

Here is the log, not super helpful. We are running the axon server within a container. If you know how to get more detailed logs, let me know and I can repost. If you look at the last 2 lines, it is saying recover-api-client (our main app) and reporting-etl-client (our springboot app) are both connected. I did initiate a begin and end event and nothing is showing in the log.

Regards,
Ed.

Listening for transport dt_socket at address: 8000
16:56:02.891 [main] INFO org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
     _                     ____
    / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
   / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
  / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
 /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
 Standard Edition                        Powered by AxonIQ

version: 4.2.4
2023-01-23 16:56:03.939  INFO 9 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer on ee3234e1bb78 with PID 9 (/opt/axonserver/axonserver.jar started by root in /opt/axonserver)
2023-01-23 16:56:03.943  INFO 9 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to default profiles: default
2023-01-23 16:56:04.332  WARN 9 --- [kground-preinit] org.springframework.web.HttpLogging      : For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
2023-01-23 16:56:10.396  INFO 9 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2023-01-23 16:56:12.347  WARN 9 --- [           main] o.f.c.internal.database.base.Database    : Flyway upgrade recommended: H2 1.4.199 is newer than this version of Flyway and support has not been tested.
2023-01-23 16:56:18.708  INFO 9 --- [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2023-01-23 16:56:19.205  INFO 9 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8024 (http) with context path ''
2023-01-23 16:56:19.210  INFO 9 --- [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 16.192 seconds (JVM running for 17.978)
2023-01-23 17:33:55.091  INFO 9 --- [pool-5-thread-5] i.a.a.logging.TopologyEventsLogger       : Application connected: recover-api, clientId = recover-api-client, context = default
2023-01-23 17:36:33.825  INFO 9 --- [ool-5-thread-11] i.a.a.logging.TopologyEventsLogger       : Application connected: reporting-etl, clientId = reporting-etl-client, context = default

Hi Ed,

Indeed, the log file does not contain any indication of the “disappearance” of events. The description of your issue has many similarities with my concern, with the only difference that in the log, I have this information which explains why my services do not receive events, but I cannot solve this problem

2023-01-20 12:16:22.929 DEBUG 871 --- [grpc-executor-2] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.118 DEBUG 871 --- [grpc-executor-4] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.268 DEBUG 871 --- [grpc-executor-4] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.269 DEBUG 871 --- [grpc-executor-4] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.117 DEBUG 871 --- [grpc-executor-1] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.270 DEBUG 871 --- [grpc-executor-1] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 
2023-01-20 12:16:23.271 DEBUG 871 --- [grpc-executor-1] i.a.a.l.TrackingEventProcessorManager    : default: Blacklisting: type: "io.shine.coreapi.messaging.tenant.TenantRegisteredEvent"
 for 

I thought this could have been the issue encountered on your side too, so we could join forces to tackle it.

Arnaud

Hey Arnaud,

I have enabled logging in my axonserver docker container and I saw the exact error you are seeing! My events are blacklisted just like yours. Let me dig around a little and see if I can find out why my events are blacklisted.

1 Like

That’s a kind of good news as it has been reproduce from another person’s project. I have tried disabling the default settings with this for my microservices consumers of the event blacklisted

axon.axonserver.disableEventBlacklisting=true

Let me know if it works for you

Hi Arnaud,

Blacklisting events is just a result of no event handler was found to handle the event. It is just the result but the cause (of the problem) is still the fact that event handlers registered are not firing to handle the event.

I had a hard time believing that it is that buggy because the axon server is not new and few people ran into this issue.

So tomorrow I will concentrate on looking at how to properly configure the event handlers for spring boot apps.

Ed.

Hi Ed,

the strange situation here in my case is that I do have handlers for the event blacklisted : it was working for quiet some months without issue.

That event have been consumed and projections built, and I noticed that it is not the case anymore, but could not spot why exactly.

I have tried this also but without success :

axon.axonserver.disableEventBlacklisting=true

If anyone have an idea where I can look at, that will be great.

Arnaud

Arnaud,

I have no idea why your code works and stopped working all of a sudden. Like I said, according to the docs, blacklisting events means no handler was found to handle the events causing blacklisting. I also found out that sometimes, the events did not get blacklisted and yet no handlers was found handling them. As in your case what causes it to stop all of a sudden, I have no idea. We used the same term “blacklist” in another way. Is it possible some of your “TenantRegistration” went amock or had errors and fail somewhere causing blacklisting?

Ed.