OpenTelemetry -- "Span ended a second time!" error

We encounter this (since we started using latest axon opentelemetry lib that’s based on otel 1.19?? not 100% sure):

We just migrated to Spring Boot 3 on JDK17, and the latest version of Axon fw.

Never encountered this exception before (when using otel 1.18 and Spring Boot 2.7.x and Axon 4.6.x), or at least never noticed it.

Any ideas?

Plz note that we have a RSocketInterceptor (because we use an RSocket handlerto send commands on ReactiveGateway). See code below.

Each rsocket message passes through this interceptor, and receives its own span.
In the interceptor we use a ‘hot’ publisher (Reactor Hot Publisher vs Cold Publisher | Vinsguru).

        public Mono<Void> fireAndForget(Payload payload) {
            return Mono.defer(
                    () -> {
                        Span span = tracer.spanBuilder("OTelRSocket fireAndForget")
                        return delegate
                                .doFinally(signalType -> span.end());


The actual sending of the command on a ReactiveCommandGateway happens in the delegate.
After the command has been sent, the interceptor ends the span.
Or at least that’s why I’d expect.
But it looks like - maybe - this interceptor has already ended the span, while axon attempts to end it again somewhat later. Maybe due to reactive/async behavior i.e. race condition.
This behavior is flaky.

Any ideas or suggestions?

PS: if you ask yourself why we use this interceptor, see Axon-tracing-opentelemetry reuses traceid - Axon Framework - Discuss.AxonIQ

Hey Christian, could you share the entire stack with me? I don’t believe it’s your code, as the glimpse I see is on the AF receiving side. But I cannot be sure without the entire callstack