Possible memory leak when doing replay using Persistent Stream

Hi everyone!

After triggering replay on Persistent Stream with simple filter condition (payloadType=“…”) Axonserver (2024.2.4) JVM heap memory consumption constantly growing and not being garbage collected. Finally JVM get OOM error.

Here is histogram of top objects before OOM (jmap -histo:live ):

num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:       4967613      198704520  java.math.BigDecimal (java.base@17.0.13)
   2:       5454305      174537760  java.util.concurrent.ConcurrentHashMap$Node (java.base@17.0.13)
   3:       4968384      119241216  java.util.Arrays$ArrayList (java.base@17.0.13)
   4:       4967578      119221872  [Lio.axoniq.axonserver.localstorage.query.ExpressionResult;
   5:       4967578       79481248  io.axoniq.axonserver.localstorage.query.result.ListExpressionResult
   6:       4967578       79481248  io.axoniq.axonserver.localstorage.query.result.NumericExpressionResult
   7:          4680       55052304  [Ljava.util.concurrent.ConcurrentHashMap$Node; (java.base@17.0.13)
   8:        527372       49947376  [B (java.base@17.0.13)
   9:        227314       15551552  [Ljava.lang.Object; (java.base@17.0.13)
  10:         19218       13100976  [J (java.base@17.0.13)
  11:        475003       11400072  java.lang.String (java.base@17.0.13)
  12:        240668        5776032  io.axoniq.axonserver.localstorage.file.k
  13:        327641        5242256  java.lang.Integer (java.base@17.0.13)
  14:        212808        5107392  java.lang.Long (java.base@17.0.13)
  15:         51431        4525928  java.lang.reflect.Method (java.base@17.0.13)
  16:         37725        4441760  java.lang.Class (java.base@17.0.13)
  17:        121386        3884352  java.util.HashMap$Node (java.base@17.0.13)
  18:         89817        3592680  io.axoniq.axonserver.localstorage.file.xref.JumpSkipIndexEntries
  19:         89817        2874144  io.axoniq.axonserver.localstorage.file.AppendOnlyList
  20:         43077        2866360  [Ljava.util.HashMap$Node; (java.base@17.0.13)
  21:         66245        2649800  java.util.LinkedHashMap$Entry (java.base@17.0.13)
  22:         19110        2407056  [I (java.base@17.0.13)
  23:         89817        2155608  io.axoniq.axonserver.localstorage.file.StandardIndexEntries
  24:         89614        2150736  org.mapdb.elsa.ElsaSerializerBase$DeserSingleton
  25:         51735        2069400  io.axoniq.axonserver.localstorage.transaction.d
  26:          2880        1935360  io.grpc.netty.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
  27:          5203        1896656  [C (java.base@17.0.13)
  28:         32515        1820840  java.util.LinkedHashMap (java.base@17.0.13)
  29:         26421        1690944  java.nio.DirectByteBuffer (java.base@17.0.13)
  30:        105404        1686464  java.util.concurrent.atomic.AtomicInteger (java.base@17.0.13)
  31:         94918        1518688  java.lang.Object (java.base@17.0.13)
  32:         91725        1467600  java.util.concurrent.atomic.AtomicReference (java.base@17.0.13)
  33:         89819        1438128  [Ljava.lang.Integer; (java.base@17.0.13)
  34:         34512        1380480  java.util.WeakHashMap$Entry (java.base@17.0.13)
  35:         57222        1373328  java.time.LocalDateTime (java.base@17.0.13)
  36:         27408        1315584  java.io.FileDescriptor (java.base@17.0.13)
  37:          2422        1288496  [Lorg.mapdb.elsa.ElsaSerializerBase$Deser;
  38:         26173        1256304  sun.nio.ch.FileChannelImpl$DefaultUnmapper (java.base@17.0.13)
  39:         25342        1216416  org.aspectj.weaver.reflect.ShadowMatchImpl

Note that there are many *ExpressionResult objects.

Client application use Axonframework 4.11.2 and configures persistent stream via axon.axonserver.persistent-streams properties.

Hope this helps to find an issue or find that i’m doing something wrong.