Axon + Testcontainers and integration tests in Gitlab CI

I am using Spring + JPA based implementation for event store and running integration tests in testcontainers for full integration flows which use entire application context.

All tests are simple Webflux Controller tests with WebTestClient.

When running int tests locally, there are no issues, everything runs fine. In Gitlab CI tests still pass fine and everything is green, but I am always observing behaviour when all tests are done then Gitlab docker instance where testcontainers + postgres is run, is shutdown after tests before the application, but the application context cannot killed, because Axon/Spring keeps trying to recover the connection. First the connection pools are all shutdown:

2021-12-16 14:01:13.803  INFO 1195 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2021-12-16 14:01:13.804  INFO 1195 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2021-12-16 14:01:13.807  INFO 1195 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
2021-12-16 14:01:13.809  INFO 1195 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2021-12-16 14:01:13.809  INFO 1195 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown initiated...
2021-12-16 14:01:13.810  INFO 1195 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown completed.
2021-12-16 14:01:13.812  INFO 1195 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
...

but then they start to recover for 30 seconds each event processor and produces massive amount of logs for such stack trace. Considering I have around 50 event processors, then I get around 4000 lines of warnings and errors after shutdown.

2021-12-16 16:02:15.465  WARN 1248 --- [dingSignSaga]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 08001
2021-12-16 16:02:15.465 ERROR 1248 --- [dingSignSaga]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-2 - Connection is not available, request timed out after 30001ms.
...
2021-12-16 16:02:45.904 ERROR 1248 --- [[scoringSaga]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-2 - Connection is not available, request timed out after 30000ms.
2021-12-16 16:02:45.904 ERROR 1248 --- [scoringViews]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : Connection to docker:49162 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2021-12-16 16:02:45.904 ERROR 1248 --- [[scoringSaga]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : Connection to docker:49162 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2021-12-16 16:02:46.466  WARN 1248 --- [seStateViews]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 08001
2021-12-16 16:02:46.467 ERROR 1248 --- [seStateViews]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-2 - Connection is not available, request timed out after 30000ms.
2021-12-16 16:02:46.467 ERROR 1248 --- [seStateViews]-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : Connection to docker:49162 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2021-12-16 16:02:46.467  WARN 1248 --- [seStateViews]-0] o.a.e.TrackingEventProcessor             : Releasing claim on token and preparing for retry in 2s
2021-12-16 16:02:46.571  WARN 1248 --- [dedEventStore-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 08001
2021-12-16 16:02:46.571 ERROR 1248 --- [dedEventStore-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-2 - Connection is not available, request timed out after 30000ms.
2021-12-16 16:02:46.571 ERROR 1248 --- [dedEventStore-0] o.h.engine.jdbc.spi.SqlExceptionHelper   : Connection to docker:49162 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2021-12-16 16:02:46.572 ERROR 1248 --- [dedEventStore-0] o.a.e.eventstore.EmbeddedEventStore      : Failed to read events from the underlying event storage

and it ends with:

2021-12-16 16:02:58.082  WARN 1248 --- [ionShutdownHook] o.a.config.DefaultConfigurer             : Timed out during shutdown phase [1073741823] after 5 seconds. Proceeding to following phase

Is there any way to gracefully shutdown Axon or kill testcontainers after application is shutdown or something else? As I mentioned then all tests pass, but the cleanup is messy :slight_smile:

I am currently not doing any special teardown in tests themselves or haven’t configured anything differently in test profiles.

Hi @vaelyr,

seems like the container where the database (event store) is running is shut down ‘too early’ and the app is still running and trying to connect to that.
Can you let us know which AF version are you using?
We had several improvements in regards to graceful shutdown!

KR,

I am using Axon BOM with version 4.5.6