[AXONIQ-6000] Missing worker for context: default

Hi,

I’m getting the following error after starting the Axon Server 4.4.3.

I tried to start with a new event store and without applications connected, but the error persists.

log

Picked up JAVA_TOOL_OPTIONS: -Xmx1g
     _                     ____
    / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
   / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
  / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
 /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
 Standard Edition                        Powered by AxonIQ
version: 4.4.3
2021-04-01 19:49:11.710  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer on axonserver-0 with PID 1 (/app/classes started by root in /)
2021-04-01 19:49:11.713  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to default profiles: default
2021-04-01 19:49:20.805  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2021-04-01 19:49:21.494  INFO 1 --- [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2021-04-01 19:49:28.813  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 4.4.3
2021-04-01 19:49:35.488  INFO 1 --- [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2021-04-01 19:49:36.793  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8024 (http) with context path '/axonserver'
2021-04-01 19:49:36.800  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 27.103 seconds (JVM running for 28.211)
2021-04-01 19:50:00.214 ERROR 1 --- [grpc-executor-3] io.grpc.internal.SerializingExecutor     : Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@63e7726c

io.axoniq.axonserver.exception.MessagingPlatformException: [AXONIQ-6000] Missing worker for context: default
	at io.axoniq.axonserver.localstorage.LocalEventStore.workers(LocalEventStore.java:198) ~[classes/:na]
	at io.axoniq.axonserver.localstorage.LocalEventStore.getFirstToken(LocalEventStore.java:430) ~[classes/:na]
	at io.axoniq.axonserver.message.event.EventDispatcher.lambda$getFirstToken$6(EventDispatcher.java:307) ~[classes/:na]
	at java.base/java.util.Optional.ifPresent(Optional.java:183) ~[na:na]
	at io.axoniq.axonserver.message.event.EventDispatcher.getFirstToken(EventDispatcher.java:306) ~[classes/:na]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172) ~[grpc-stub-1.27.2.jar:1.27.2]
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

axon server.properties

    # Copyright (c) 2018 by Axoniq B.V. - All rights reserved
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    # HTTP server port
    # Axon Server settings
    #   The values shown below are the defaults.
    # The HTTP port for GUI and REST API
    # server.port=8024
    # Storage locations
    axoniq.axonserver.event.storage=/eventdata
    axoniq.axonserver.snapshot.storage=/eventdata
    axoniq.axonserver.controldb-path=/data
    axoniq.axonserver.pid-file-location=/data
    logging.file=/data/axonserver.log
    logging.file.max-history=10
    logging.file.max-size=10MB
    logging.level.io.axoniq.axonserver=INFO 
    # gRPC port for axonserver platform
    # axoniq.axonserver.port=8124
    # Node name of this axonserver platform node, if not set defaults to the hostname.
    # axoniq.axonserver.name=
    # Hostname of this node as communicated to clients, defaults to the result of hostname command
    # axoniq.axonserver.hostname=
    # Domain of this node as communicated to clients. Optional, if set will be appended to the hostname in communication with clients.
    # axoniq.axonserver.domain=
    # Cluster enabled
    # axoniq.axonserver.cluster.enabled=false
    # gRPC port for communication between messing platform nodes
    # axoniq.axonserver.internal-port=8224
    # Hostname as communicated to other nodes of the cluster. Defaults to hostname.
    # axoniq.axonserver.internal-hostname=
    # Domain as communicated to other nodes of the cluster. Optional, if not set, it will use the domain value.
    # axoniq.axonserver.internal-domain=
    # Delay before the first run of the connection checker (in ms.)
    # axoniq.axonserver.cluster.connection-check-delay=1000
    # Interval between each run of the connection checker (in ms.)
    # axoniq.axonserver.cluster.connection-check-interval=1000
    # Delay before the first run of the rebalancer (in seconds) [Enterprise edition only]
    # axoniq.axonserver.cluster.rebalance-delay=7
    # Interval between each run of the rebalancer (in seconds) [Enterprise edition only]
    # axoniq.axonserver.cluster.rebalance-interval=15
    # SSL enabled for gRPC servers
    # axoniq.axonserver.ssl.enabled=false
    # axoniq.axonserver.ssl.cert-chain-file=
    # axoniq.axonserver.ssl.internal-cert-chain-file=
    # axoniq.axonserver.ssl.private-key-file=
    # Access control active
    # axoniq.axonserver.accesscontrol.enabled=false
    # Milliseconds that authenticated tokens will be cached
    # axoniq.axonserver.accesscontrol.cache-ttl=30000
    # Token to add to AxonHub internal cluster message
    # axoniq.axonserver.accesscontrol.internal-token=
    # Token expected from client requests [Developer edition only]
    # axoniq.axonserver.accesscontrol.token=
    # Initial number of permits granted in communication between AxonHub nodes.
    # axoniq.axonserver.command-flow-control.initial-permits=10000
    # Additional number of permits granted in communication between AxonHub nodes.
    # axoniq.axonserver.command-flow-control.new-permits=10000
    # Threshold at which the node will send another grant of newPermits to the connected platform node.
    # axoniq.axonserver.command-flow-control.threshold=1000
    # Initial number of permits granted in communication between AxonHub nodes.
    # axoniq.axonserver.query-flow-control.initial-permits=10000
    # Additional number of permits granted in communication between AxonHub nodes.
    # axoniq.axonserver.query-flow-control.new-permits=10000
    # Threshold at which the node will send another grant of newPermits to the connected platform node.
    # axoniq.axonserver.query-flow-control.threshold=1000
    axoniq.axonserver.heartbeat.enabled=true

Hi Rodrigo,
From the logging, it looks like you are using the docker image. Can you restart the image with an additional logging option, logging.level.io.axoniq.axonserver.localstorage=TRACE. The easiest way to do this is to add this to the JAVA_TOOL_OPTIONS, for instance:

docker run -it -p 8024:8024 -e JAVA_TOOL_OPTIONS="-Xmx1g -Dlogging.level.io.axoniq.axonserver.localstorage=TRACE" axoniq/axonserver:4.4.3

This should provide more logging information on the initialization of the event store.

Marc

Hi Marc,

This is the log with TRACE enabled for package io.axoniq.axonserver.localstorage.

Picked up JAVA_TOOL_OPTIONS: -Xmx1g -Dlogging.level.io.axoniq.axonserver.localstorage=TRACE
     _                     ____
    / \   __  _____  _ __ / ___|  ___ _ ____   _____ _ __
   / _ \  \ \/ / _ \| '_ \\___ \ / _ \ '__\ \ / / _ \ '__|
  / ___ \  >  < (_) | | | |___) |  __/ |   \ V /  __/ |
 /_/   \_\/_/\_\___/|_| |_|____/ \___|_|    \_/ \___|_|
 Standard Edition                        Powered by AxonIQ

version: 4.4.3
2021-04-05 01:09:30.645  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Starting AxonServer on axonserver-0 with PID 1 (/app/classes started by root in /)
2021-04-05 01:09:30.648  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : No active profile set, falling back to default profiles: default
2021-04-05 01:09:38.563  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8024 (http)
2021-04-05 01:09:39.258  INFO 1 --- [           main] A.i.a.a.c.MessagingPlatformConfiguration : Configuration initialized with SSL DISABLED and access control DISABLED.
2021-04-05 01:09:45.941  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Axon Server version 4.4.3
2021-04-05 01:09:46.853 DEBUG 1 --- [           main] i.a.a.localstorage.LocalEventStore       : default: initializing
2021-04-05 01:09:51.880  INFO 1 --- [           main] io.axoniq.axonserver.grpc.Gateway        : Axon Server Gateway started on port: 8124 - no SSL
2021-04-05 01:09:53.040  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8024 (http) with context path '/axonserver'
2021-04-05 01:09:53.046  INFO 1 --- [           main] io.axoniq.axonserver.AxonServer          : Started AxonServer in 24.298 seconds (JVM running for 25.091)
2021-04-05 01:10:49.356 ERROR 1 --- [grpc-executor-3] io.grpc.internal.SerializingExecutor     : Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@6cee80b9

io.axoniq.axonserver.exception.MessagingPlatformException: [AXONIQ-6000] Missing worker for context: default
	at io.axoniq.axonserver.localstorage.LocalEventStore.workers(LocalEventStore.java:198) ~[classes/:na]
	at io.axoniq.axonserver.localstorage.LocalEventStore.getFirstToken(LocalEventStore.java:430) ~[classes/:na]
	at io.axoniq.axonserver.message.event.EventDispatcher.lambda$getFirstToken$6(EventDispatcher.java:307) ~[classes/:na]
	at java.base/java.util.Optional.ifPresent(Optional.java:183) ~[na:na]
	at io.axoniq.axonserver.message.event.EventDispatcher.getFirstToken(EventDispatcher.java:306) ~[classes/:na]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172) ~[grpc-stub-1.27.2.jar:1.27.2]
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) ~[grpc-api-1.27.2.jar:1.27.2]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.27.2.jar:1.27.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.27.2.jar:1.27.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

I’m running Axon Server in an AKS at Azure using azure-disk as storage.

Hi Marc,

Sorry, the error occurred because of a wrong volumeMounts section from the statefulset template.

Everything is working now.