Hi everyone!
After triggering replay on Persistent Stream with simple filter condition (payloadType=“…”) Axonserver (2024.2.4) JVM heap memory consumption constantly growing and not being garbage collected. Finally JVM get OOM error.
Here is histogram of top objects before OOM (jmap -histo:live ):
num #instances #bytes class name (module)
-------------------------------------------------------
1: 4967613 198704520 java.math.BigDecimal (java.base@17.0.13)
2: 5454305 174537760 java.util.concurrent.ConcurrentHashMap$Node (java.base@17.0.13)
3: 4968384 119241216 java.util.Arrays$ArrayList (java.base@17.0.13)
4: 4967578 119221872 [Lio.axoniq.axonserver.localstorage.query.ExpressionResult;
5: 4967578 79481248 io.axoniq.axonserver.localstorage.query.result.ListExpressionResult
6: 4967578 79481248 io.axoniq.axonserver.localstorage.query.result.NumericExpressionResult
7: 4680 55052304 [Ljava.util.concurrent.ConcurrentHashMap$Node; (java.base@17.0.13)
8: 527372 49947376 [B (java.base@17.0.13)
9: 227314 15551552 [Ljava.lang.Object; (java.base@17.0.13)
10: 19218 13100976 [J (java.base@17.0.13)
11: 475003 11400072 java.lang.String (java.base@17.0.13)
12: 240668 5776032 io.axoniq.axonserver.localstorage.file.k
13: 327641 5242256 java.lang.Integer (java.base@17.0.13)
14: 212808 5107392 java.lang.Long (java.base@17.0.13)
15: 51431 4525928 java.lang.reflect.Method (java.base@17.0.13)
16: 37725 4441760 java.lang.Class (java.base@17.0.13)
17: 121386 3884352 java.util.HashMap$Node (java.base@17.0.13)
18: 89817 3592680 io.axoniq.axonserver.localstorage.file.xref.JumpSkipIndexEntries
19: 89817 2874144 io.axoniq.axonserver.localstorage.file.AppendOnlyList
20: 43077 2866360 [Ljava.util.HashMap$Node; (java.base@17.0.13)
21: 66245 2649800 java.util.LinkedHashMap$Entry (java.base@17.0.13)
22: 19110 2407056 [I (java.base@17.0.13)
23: 89817 2155608 io.axoniq.axonserver.localstorage.file.StandardIndexEntries
24: 89614 2150736 org.mapdb.elsa.ElsaSerializerBase$DeserSingleton
25: 51735 2069400 io.axoniq.axonserver.localstorage.transaction.d
26: 2880 1935360 io.grpc.netty.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
27: 5203 1896656 [C (java.base@17.0.13)
28: 32515 1820840 java.util.LinkedHashMap (java.base@17.0.13)
29: 26421 1690944 java.nio.DirectByteBuffer (java.base@17.0.13)
30: 105404 1686464 java.util.concurrent.atomic.AtomicInteger (java.base@17.0.13)
31: 94918 1518688 java.lang.Object (java.base@17.0.13)
32: 91725 1467600 java.util.concurrent.atomic.AtomicReference (java.base@17.0.13)
33: 89819 1438128 [Ljava.lang.Integer; (java.base@17.0.13)
34: 34512 1380480 java.util.WeakHashMap$Entry (java.base@17.0.13)
35: 57222 1373328 java.time.LocalDateTime (java.base@17.0.13)
36: 27408 1315584 java.io.FileDescriptor (java.base@17.0.13)
37: 2422 1288496 [Lorg.mapdb.elsa.ElsaSerializerBase$Deser;
38: 26173 1256304 sun.nio.ch.FileChannelImpl$DefaultUnmapper (java.base@17.0.13)
39: 25342 1216416 org.aspectj.weaver.reflect.ShadowMatchImpl
Note that there are many *ExpressionResult objects.
Client application use Axonframework 4.11.2 and configures persistent stream via axon.axonserver.persistent-streams properties.
Hope this helps to find an issue or find that i’m doing something wrong.