Event persisted but event handlers not processed


I see axonConnectionManager trying to release unused connections but after that it started restarting not sure whats happening
it basically break whole business logic.

WARN 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 1s"
[ends.service]-0] o.a.e.TrackingEventProcessor             : Error occurred. Starting retry mode."

workspaceId=3d15aa1c-7816-49c7-af21-e698e953f919
workspace name=connect-with-friends-new-new
context-name=friends
Application binding name=aglet-backend-friends

[
  {
    "textPayload": "2023-01-24 06:29:07.781  INFO 1 --- [ult-executor-95] i.a.a.c.query.impl.QueryChannelImpl      : QueryChannel for context 'friends@connect-with-friends-new-new' connected, 7 registrations resubscribed",
    "insertId": "fzxt6jiw3e28hq41",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:29:07.781344598Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "textPayload": "2023-01-24 06:29:07.773  INFO 1 --- [ult-executor-95] i.a.a.c.command.impl.CommandChannelImpl  : CommandChannel for context 'friends@connect-with-friends-new-new' connected, 16 command handlers registered",
    "insertId": "o6abh4fhfx9rxcpe",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:29:07.773588626Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "insertId": "mlj1zg8kkozzkzj2",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:29:06.963326689Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "textPayload": "io.grpc.StatusRuntimeException: CANCELLED: Query stream inactivity for 1@backend-friends-deployment-85dc69f668-f5mlv.ecc05b7d-f973-4d7f-9fde-6d7f4ebeb78f\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]",
    "insertId": "8cgqtls59qjiq72w",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "namespace_name": "aglet",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:29:06.963229979Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "insertId": "anxtm88nsn8wwocu",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:29:06.963218776Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "textPayload": "2023-01-24 06:29:06.962  INFO 1 --- [ult-executor-95] i.a.a.c.query.impl.QueryChannelImpl      : Error on QueryChannel for context friends@connect-with-friends-new-new",
    "insertId": "betp5tdt20asveki",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:29:06.963157735Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:10.215671664Z"
  },
  {
    "textPayload": "2023-01-24 06:29:02.831  INFO 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Fetched token: IndexTrackingToken{globalIndex=74} for segment: Segment[0/0]",
    "insertId": "5p1s9xrtds3nen3r",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:29:02.831298127Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:02.830  INFO 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Fetched token: IndexTrackingToken{globalIndex=74} for segment: Segment[0/0]",
    "insertId": "tgyv4z0arndl0chz",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:29:02.830485849Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:02.089  INFO 1 --- [ult-executor-95] i.a.a.c.query.impl.QueryChannelImpl      : QueryChannel for context 'friends@connect-with-friends-new-new' connected, 7 registrations resubscribed",
    "insertId": "oupcb7zt0mxm1hy0",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:29:02.090275666Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:02.077  INFO 1 --- [ult-executor-91] i.a.a.c.command.impl.CommandChannelImpl  : CommandChannel for context 'friends@connect-with-friends-new-new' connected, 16 command handlers registered",
    "insertId": "50rt0d2k4fp9yemo",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:29:02.077693695Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.852  INFO 1 --- [c69f668-f5mlv-1] i.a.a.connector.impl.ControlChannelImpl  : Connected instruction stream for context 'friends@connect-with-friends-new-new'. Sending client identification",
    "insertId": "89zrynrl46gijd5d",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.852950993Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "insertId": "ogcasi6zlny1ir3p",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:29:01.819943459Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[na:na]\n\tat java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[na:na]\n\tat io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\t... 1 common frames omitted",
    "insertId": "txtkjgjmm0fiugyw",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.819734605Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "insertId": "6mqpywwe59ocnh6h",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.819721973Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.816  INFO 1 --- [ult-executor-95] i.a.a.c.query.impl.QueryChannelImpl      : Error on QueryChannel for context friends@connect-with-friends-new-new",
    "insertId": "1g669polro41tsiw",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.819663532Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.810  INFO 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Released claim",
    "insertId": "1nvbittbx7zbvyah",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:29:01.811199453Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.809  INFO 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Released claim",
    "insertId": "e2f3cnqf527hrsbj",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.810169712Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.805  WARN 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 1s",
    "insertId": "py6wvq2luin4ryjn",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:29:01.805411710Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "insertId": "v7rnp1bh0w3tjz2d",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803913180Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "org.axonframework.axonserver.connector.AxonServerException: The Event Stream has been closed, so no further events can be retrieved\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.peekNullable(EventBuffer.java:178) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.hasNextAvailable(EventBuffer.java:144) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:418) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:318) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1145) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.cleanUp(TrackingEventProcessor.java:1340) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1317) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]\nCaused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\t... 1 common frames omitted\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[na:na]\n\tat java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[na:na]\n\tat io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\t... 1 common frames omitted",
    "insertId": "ujq3f9y64njc84md",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803560307Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  }
]
[
{
    "insertId": "6mc9s51s2qnjlods",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803536104Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.802  WARN 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Error occurred. Starting retry mode.",
    "insertId": "3g5gg9a5l3t6l9fo",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803530478Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.802  WARN 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 1s",
    "insertId": "acd4jimxa4s25x0g",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803521740Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "insertId": "1ciwjoieaqitoeb0",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803515689Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "org.axonframework.axonserver.connector.AxonServerException: The Event Stream has been closed, so no further events can be retrieved\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.peekNullable(EventBuffer.java:178) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.hasNextAvailable(EventBuffer.java:144) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:418) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:318) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1145) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.cleanUp(TrackingEventProcessor.java:1340) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1317) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]\nCaused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\t... 1 common frames omitted\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[na:na]\n\tat java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[na:na]\n\tat io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]\n\t... 1 common frames omitted",
    "insertId": "hsnw3wul50j8ihz5",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803089081Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "insertId": "9zkfap4hmro2a90m",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:29:01.803076224Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:29:01.802  WARN 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Error occurred. Starting retry mode.",
    "insertId": "2kx8ksvz06tkrby7",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:29:01.802971319Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:29:05.223700834Z"
  },
  {
    "textPayload": "2023-01-24 06:26:51.683  INFO 1 --- [ult-executor-91] i.a.a.c.command.impl.CommandChannelImpl  : CommandChannel for context 'friends@connect-with-friends-new-new' connected, 16 command handlers registered",
    "insertId": "7sodbl2wfg05ahzp",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:26:51.684275739Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "textPayload": "2023-01-24 06:26:51.675  INFO 1 --- [ult-executor-91] i.a.a.c.query.impl.QueryChannelImpl      : QueryChannel for context 'friends@connect-with-friends-new-new' connected, 7 registrations resubscribed",
    "insertId": "9p6f3honrclmy33r",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev",
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:26:51.676537198Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "insertId": "agwz58aa9gm7lwvi",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:50.968443487Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "textPayload": "io.grpc.StatusRuntimeException: CANCELLED: Query stream inactivity for 1@backend-friends-deployment-85dc69f668-f5mlv.c23f333e-570b-4116-b75d-88d4d338f516\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]",
    "insertId": "tr0o8xwbpqevypv7",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:50.968319823Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "insertId": "gii098knw8l6unfn",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:50.968307494Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "textPayload": "2023-01-24 06:26:50.967  INFO 1 --- [ult-executor-91] i.a.a.c.query.impl.QueryChannelImpl      : Error on QueryChannel for context friends@connect-with-friends-new-new",
    "insertId": "6ycgozk9azt621ht",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:26:50.968236493Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:55.222588377Z"
  },
  {
    "textPayload": "2023-01-24 06:26:46.755  INFO 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Fetched token: IndexTrackingToken{globalIndex=74} for segment: Segment[0/0]",
    "insertId": "8ihtmufpyq59tr16",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:26:46.756257575Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:46.755  INFO 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Fetched token: IndexTrackingToken{globalIndex=74} for segment: Segment[0/0]",
    "insertId": "zv9sd45v109sdeov",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:26:46.755756020Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:46.120  INFO 1 --- [ult-executor-91] i.a.a.c.query.impl.QueryChannelImpl      : QueryChannel for context 'friends@connect-with-friends-new-new' connected, 7 registrations resubscribed",
    "insertId": "taar1lqwhqomsfn1",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:26:46.120497429Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:46.104  INFO 1 --- [ult-executor-91] i.a.a.c.command.impl.CommandChannelImpl  : CommandChannel for context 'friends@connect-with-friends-new-new' connected, 16 command handlers registered",
    "insertId": "d5b0wtloealdhh0t",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:26:46.105408956Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.760  INFO 1 --- [c69f668-f5mlv-0] i.a.a.connector.impl.ControlChannelImpl  : Connected instruction stream for context 'friends@connect-with-friends-new-new'. Sending client identification",
    "insertId": "rc7hkw2dn8p5kskm",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:26:45.760552816Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.731  INFO 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Released claim",
    "insertId": "rnopjkk58n74t6zj",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:26:45.731875624Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.729  INFO 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Released claim",
    "insertId": "qenvy3sgg5kstnwf",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:26:45.729460494Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  }
]
[
{
    "insertId": "h6nv7kgskre9xw28",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:45.726930900Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: max_age\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]",
    "insertId": "5jco797722wqkzvv",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:26:45.726796563Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/app": "backend-friends",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "insertId": "q37hwqvmj2cxtdk8",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends"
      }
    },
    "timestamp": "2023-01-24T06:26:45.726787463Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.726  INFO 1 --- [ult-executor-91] i.a.a.c.query.impl.QueryChannelImpl      : Error on QueryChannel for context friends@connect-with-friends-new-new",
    "insertId": "y29sckf0zew2yjhq",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "aglet-cwf-dev-cluster",
        "namespace_name": "aglet",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:26:45.726751322Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.725  WARN 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 1s",
    "insertId": "owe3riwgir8i1inr",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "location": "us-central1-a",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:45.726136985Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "insertId": "ga5ukagpoom4rkx1",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster"
      }
    },
    "timestamp": "2023-01-24T06:26:45.725837744Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "org.axonframework.axonserver.connector.AxonServerException: The Event Stream has been closed, so no further events can be retrieved\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.peekNullable(EventBuffer.java:178) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.hasNextAvailable(EventBuffer.java:144) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:418) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:318) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1145) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.cleanUp(TrackingEventProcessor.java:1340) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1317) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]\nCaused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: max_age\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\t... 1 common frames omitted",
    "insertId": "0c7mjycstlq60i66",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:26:45.725629060Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "insertId": "5u3ghcr9rildxt2h",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "backend-friends",
        "namespace_name": "aglet",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a",
        "project_id": "aglet-cwf-dev",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv"
      }
    },
    "timestamp": "2023-01-24T06:26:45.725609864Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.725  WARN 1 --- [agaProcessor]-0] o.a.e.TrackingEventProcessor             : Error occurred. Starting retry mode.",
    "insertId": "72y1tktup5c6ub3b",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "project_id": "aglet-cwf-dev",
        "container_name": "backend-friends",
        "cluster_name": "aglet-cwf-dev-cluster",
        "location": "us-central1-a"
      }
    },
    "timestamp": "2023-01-24T06:26:45.725546710Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "2023-01-24 06:26:45.716  WARN 1 --- [ends.service]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 1s",
    "insertId": "ed8ba8y28gwp6eni",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "aglet",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:26:45.716820856Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "insertId": "53uqn9n3l3r7oqt6",
    "jsonPayload": {},
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "namespace_name": "aglet",
        "container_name": "backend-friends",
        "location": "us-central1-a",
        "cluster_name": "aglet-cwf-dev-cluster",
        "project_id": "aglet-cwf-dev"
      }
    },
    "timestamp": "2023-01-24T06:26:45.716360276Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  },
  {
    "textPayload": "org.axonframework.axonserver.connector.AxonServerException: The Event Stream has been closed, so no further events can be retrieved\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.peekNullable(EventBuffer.java:178) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.axonserver.connector.event.axon.EventBuffer.hasNextAvailable(EventBuffer.java:144) ~[axon-server-connector-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processBatch(TrackingEventProcessor.java:418) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:318) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$TrackingSegmentWorker.run(TrackingEventProcessor.java:1145) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.cleanUp(TrackingEventProcessor.java:1340) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat org.axonframework.eventhandling.TrackingEventProcessor$WorkerLauncher.run(TrackingEventProcessor.java:1317) ~[axon-messaging-4.6.1.jar:4.6.1]\n\tat java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]\nCaused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: max_age\n\tat io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.45.1.jar:1.45.1]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.45.1.jar:1.45.1]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]\n\t... 1 common frames omitted",
    "insertId": "f6ibu6zbucc1imst",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "aglet-cwf-dev",
        "location": "us-central1-a",
        "pod_name": "backend-friends-deployment-85dc69f668-f5mlv",
        "cluster_name": "aglet-cwf-dev-cluster",
        "container_name": "backend-friends",
        "namespace_name": "aglet"
      }
    },
    "timestamp": "2023-01-24T06:26:45.716180599Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app": "backend-friends",
      "k8s-pod/pod-template-hash": "85dc69f668",
      "k8s-pod/date": "2023-01-24_03-32-53",
      "compute.googleapis.com/resource_name": "gke-aglet-cwf-dev-cl-aglet-cwf-dev-po-6546d2c1-nms8"
    },
    "logName": "projects/aglet-cwf-dev/logs/stdout",
    "receiveTimestamp": "2023-01-24T06:26:50.219249039Z"
  }
]

Today I see those errors disappeared, as usual all the events are persisted but none of the event handlers are processed, so projection is not updated. Can you please look into this?
What is the best alternative to reach out to customer support when we get stuck on issue like this?

Hi @Arun ,

I see invalid sequence number 0 for aggregate...expected 2 in the stack trace. It might indicate that two concurrent transactions are trying to write to the same aggregate event stream simultaneously.

BTW, the event processor/handler seems to be in Error/Retry mode for some reason. Do you see any exceptions on the application side? For example, the Event handler is not able to process event and store to projection tables for some reason.

  1. What Axon Server are you using: Axon Server SE, Axon Server EE, or AxonIQ CLoud? AxonIQ Cloud most probably, but just to be sure.
  2. How many instances of application(s) that are handling commands and events do you have?
  3. What type of event processor is it? Tracking, Subscribing?
  4. Do you have any specific configuration for the Event Processors/Handlers? segments or threads?
  5. Have you additionally configured axon.axonserver.command-threads=10 or axon.axonserver.query-threads=10 within your application.properties/yml? By default, it is 10, if you have not configured it explicitly. This way you can locate a greater number of threads for command/query handling to minimize contention.
  6. Is there any proxy in between applications and Axon Server?
  7. Can you share (application) machine characteristics, please? CPU, Memory available? JVM (xmx)?

Best,
Ivan

Hi Ivan
We fired the same request twice, first time we got 500 as status code, so we thought the even is not persisted, so we fired the same request second time so we got that invalid sequence number 0 for aggregate...expected 2 in the stacktrace. you can ignore that message. The stack trace above the invalid sequence number 0 for aggregate...expected 2 is what we got first time.
Even we are not sure why and how it has gone to Error/Retry mode.

  1. What Axon Server are you using: Axon Server SE, Axon Server EE, or AxonIQ CLoud? AxonIQ Cloud most probably, but just to be sure.
    Ans: We are using AxonIQ cloud
  2. How many instances of application(s) that are handling commands and events do you have?
    Ans: Only one instance is running currently as it is development environment.
  3. What type of event processor is it? Tracking, Subscribing?
    Ans: we are using spring annotation @EventHandler for processing event, so, not sure about the type.
  4. Do you have any specific configuration for the Event Processors/Handlers? segments or threads?
    Ans: We haven’t configured anything other than
axon.serializer.general=jackson
axon.serializer.events=jackson
axon.serializer.messages=XStream

so we are using default configuration

  1. Have you additionally configured axon.axonserver.command-threads=10 or axon.axonserver.query-threads=10 within your application.properties/yml? By default, it is 10, if you have not configured it explicitly. This way you can locate a greater number of threads for command/query handling to minimize contention.
    Ans: No, we haven’t customized any such configuration.
  2. Is there any proxy in between applications and Axon Server?
    Ans: There is no proxy involved between service and axonIQ
  3. Can you share (application) machine characteristics, please? CPU, Memory available? JVM (xmx)?
    Ans:

Please let me know if you need any other information.
Thanks for looking into it.
We are blocked as of now. Hope we get to the bottom of this issue…

From today we are not getting any error message in our application log but the @EventHandlers are not processed, so it is not persisted in database.

@method annotated with @EventSourcingHandler is procesed but @EventHandler is not
code snipper for your reference

    @EventSourcingHandler
    public final void on(final ReceivedFriendRequestEvent event) {
        receivedFriendRequests.add(event.getSenderPlayerId());
    }

    @EventHandler
    public void on(final ReceivedFriendRequestEvent event) {
        repository.save(new ReceivedFriendshipRequest(event.getPlayerId(), event.getSenderPlayerId()));
    }

Note: there is no application code change, all the functionality were working fine couple of weeks back.

Alright, thanks for the info.
Everything seems to be OK from the configuration point of view.
You are using the default configuration in most places.

It is strange that you don’t have any exceptions.
I will speak with our cloud team and try to debug it from our side.
I will keep you in the loop, with our best effort.

It looks like the event handlers/processor is not registered at all, or for some reason stuck in Error/Retry mode (which is strange, because there is no exception thrown from your event handler methods). What these event handlers are doing? Are they updating some projections/tables? If so, it could be because of connectivity with the database the exception will propagate into Error/Retry mode. This can happen without you changing anything in the source code.

BTW, these EventProcessors/handlers are by default of Tracking nature. It means that it maintains a token/position of the last processed event from the event store. It is a token table in your system. I am curious what is the content of it.

Hello @Arun! The framework should recover automatically from issues like this. Obviously, in your case, it didn’t. Would it be possible for you to provide a thread dump to me? I would love to analyze it and see where the event processor is stuck.
Is the token table for that processor still updated? Or does the token have a timestamp stuck at a previous time? And does the application recover after restarting it? (Please only restart after taking the thread dump).

Thanks for all the information so far already!

I think we identified the cause.
Essentially, the context was recreated (deleted and created), but without deleting the token store.

Once you end up in a situation where your number of events is 43 in the event store context, and the latest token position (of event handler/processor) is 74, event handlers can not read new events because 74 is greater than 43.

We will send you a full description of the problem and solution over our official support channel as it might include sensitive data.

Best,
Ivan

Thank you so much @Ivan_Dugalic for investigating the issue and educating us. Your timely help is very much appreciated, It actually unblocked us.

1 Like