Hello, after setting up open telemetry in our axon project (running axon 4.9.2 on SB 3.1), we have found that the StreamingEventProcessor.process() span be unable to close by itself.
After finishing the operation within the event, the tracking token update spans are periodically fired until the distributedInSameTraceTimeLimit is reached.
This issue can be mitigated by instantiating up the EventProcessorSpanFactory as DefaultEventProcessorSpanFactory.builder().spanFactory(configuration.spanFactory()).disableBatchTrace(true).build()), but this comes at a cost of losing the information on what is happening inside the events.
Is this kind of behavior expected when running batch trace or, if not, would someone on this board be able to provide me with hints on what we might be doin wrong on our axon implementation?
It’s definitely not the intended result from the tracing in Axon Framework. I rechecked the code, but the Span is properly closed at all times.
What I can see in your image is that it’s not the StreamingEventProcessor.process to which the token span update is anchored, right?
To properly diagnose this, I will need to full chain of the calls, from the start, retaining the framework span names but blinding out your company’s information. I also need to know if you’re on a TrackingEventProcessor or PooledStreamingEventProcessor. I would also like to know the exact configuration that causes it, aside from the one that doesn’t cause it in your message up here.
And, last but not least, what you are using as an event store.
The span in question that keeps appearing actually comes from Spring, not Axon Framework. So it seems a span somewhere is leaking to a thread, remaining there as parent for all new spans.
I will investigate this as soon as I have more information from you
