Open Telemetry Instrumentation for Non java-agent Instrumented Spring Boot Apps

Axon Framework
, , ,
1

Hello,

I have a fleet of axon/spring boot applications built on axon-framework 4.10.0 and spring boot (webflux - 3.3.2) whose observability configuration looks like so…

    // SPRING BOOT - OBSERVABILITY
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("io.micrometer:micrometer-registry-prometheus")
    implementation("io.micrometer:micrometer-tracing-bridge-otel")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")

    // AXON FRAMEWORK - OBSERVABILITY
    implementation("org.axonframework:axon-tracing-opentelemetry:4.10.0")

My current challenge is that Message Handlers do not inherit traceId's and spans from the parent/current context. This forces us to fly blind when comes to critical operations. Here’s an example log…

// Controller Layer Log
2024-08-11T19:38:02.801+03:00  INFO 17779 --- [NOTIFICATIONS-SERVICE] [     parallel-1] [c0ec3ed4e94d45c89681d1c0a6c24a07-4b2b4117c49ddd02] i.t.c.notifications.api.SMSController    : Sending sms

// Event Handler Layer Log
2024-08-11T19:38:03.458+03:00  INFO 17779 --- [NOTIFICATIONS-SERVICE] [ions.service]-0] [00000000000000000000000000000000-0000000000000000] i.t.c.n.service.ATSMSService$Companion   : Dispatching SMS

I can infer that your tooling works best when the otel java agent is available on the JVM. I strongly prefer not to have the agent due to the production environment’s restrictions.

The documentation’s directives do not apply and I would appreciate some pointers on instrumenting otel/axon apps without the java agent.

Thank you.

2

I found a solution to the problem…

Turns out you have to explicitly inject the tracer and context propagator into the OpenTelemetrySpanFactory and apply the configuration across all apps…

import io.opentelemetry.api.trace.Tracer
import io.opentelemetry.context.propagation.TextMapPropagator
import org.axonframework.tracing.SpanFactory
import org.axonframework.tracing.opentelemetry.OpenTelemetrySpanFactory
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration

@Configuration
class AxonConfig {

    @Bean
    fun spanFactory(tracer: Tracer, contextPropagator: TextMapPropagator): SpanFactory =
        OpenTelemetrySpanFactory
            .builder()
            .tracer(tracer)
            .contextPropagators(contextPropagator)
            .build()
}