This is driving me crazy, when we have process intensive processes we more often than not got stuck with connections in situations like this:
Pool stats (total=10, active=10, idle=0, waiting=22)
While debugging the possible culprits I ended up on this:
public static class NoOpCloseHandler implements ConnectionCloseHandler {
@Override
public void close(Connection connection) {
}
@Override
public void commit(Connection connection) throws SQLException {
connection.commit();
}
}
As you can see that close doe nothing, so I’m supposing this causes the wrapped-around connection to be open and cause my problem? I do not know this exactly because that code is buried deep down axon I couldn’t find a way to change the way axon is configured in order to change it.
Basically what we have is a hikari connection wrapping a pg connection, that it seems is being wrapped by axon.
Can somebody explain what’s happening here and why axon don;t want connections to be closed?
OK, I rushed to ask this without investigating before (frustration and stress of things not working as expected) so bare with me. I did found tis answer from Allard Buijze [1[
your ConnectionProvider should return a connection that doesn’t close when close() is called, because the connection is “managed” by the outer transaction. In fact, the TransactionManager will probably already open a connection and start a transaction on it. Your ConnectionProvider should check for existing transactions and return the underlying connection is there is one. Otherwise, it can return a “regular” connection, which is closed when “close()” is is called.
OK, I understand axon doesen;t close the connections itself now. But then Allard goes on saying
Have a look at the UnitOfWorkAwareConnectionProviderWrapper. Probably, you would want to use this class as your ConnectionProvider. Then use that same instance (of UnitOfWorkAwareConnectionProviderWrapper) to get the connection to start the transaction. It will register the connection with the current Unit of Work and reuse that connection for all activity in the unit of work, only closing it when the commit was done.
Now AFAICS we are using a (extended version of) DefaultEventEntryStore that uses this UnitOfWorkAwareConnectionProviderWrapper by default, but the fact is we just ran out of connections very quickly.
I also found this post [2] from one Antonio Mota (!) that describes the same problem so either the problem resurfaced again with other causes or it was masked then by tweaking database connections but never actually solved.
We are under great pressure here (aren’t we all, right?), if somebody has any pointers please do tell.
which version of Axon are you currently running on? Looking at the classes you refer to, it seems 2.x, but which version exactly?
The code of the Axon 2.4.x branch shows that all connections are closed after the query has been executed. The UnitOfWorkAwareConnectionProviderWrapper class is an optimization that prevents multiple connections from being used in the scope of a single unit of work. It returns a Connection with a “NoOp” close() implementation, and will actually close the connection when the unit of work is cleaned up.
Do you also see active (or blocked / deadlocked) threads in your application? Or are there just open connections dangling around?