Transactional REQUIRES_NEW considered harmful — Spring/Java transaction handling pitfalls

Databases and Transactions are great, Spring is great, Kotlin/Java are great (ofc. Kotlin is better 😉), simplifying transaction-handling is great. But as we all know, simplifying things almost always creates some new pitfalls if the things that were simplified aren’t fully understood.

The Issue

The Spring @Transactional Annotation is a powerful tool to make transaction-management easy for developers.

What is Propagation.REQUIRES_NEW?

Often people think of this propagation setting as “magically tells the database to make a nested transaction”. However this mental model is wrong — many Databases don’t even have nested transactions, so something else needs to happen.

Transactional and Transactional(REQUIRES_NEW) interacting with the Connection-Pool. “Connection 2” is just an example of any currently idle connection.

Deadlocks on all Sides

Quick recap on deadlocks: “[…] a state in which each member of a group waits for another member, including itself, to take action […]” from [2]

  • Now each thread wants to acquire another connection and thereby waits for the other threads to release them.
  • The other threads to the same, so they would only release a connection after they got an additional one.
  • Then the Java-Process (same thread) has opened another connection (via a call to a @Transactional(propagation = Propagation.REQUIRES_NEW) annotated method)
  • In this second connection, it tries to insert something in a user-preferences table and use a foreign-key to the just inserted user.
  • When the inner method is left and Spring tries to flush the “inner”-transaction (second-transaction is more accurate), the database will block.
  • The database waits for the first connection to complete (commit) before it lets the second one commit.
  • This is completely valid from the database’s point of view, since those 2 connections are completely independent.
  • However the Java-side introduced a blocking dependency which runs in the opposite direction (the inner method won’t finish/return, so the calling method of course can’t finish either).

Reproduction

If you want to code/play/… along, here is the example repository with the code.

seq 1 8 | xargs -I $ -n1 -P10  curl "http://localhost:8080/nested"

Identifying The Issue

What can we do if we think our application has this problem? It may just be a feeling, but how can we be sure?

Shortened Stacktrace showing 2 threads stuck at datasource.getConnection()
Pool-status showing us that lots of connections are waiting (requested from the pool, but could not be provided yet).

Possible Solutions

There are basically 3 ways this can be fixed:

  • Serializing the 2 transactions. Instead of having an inner transaction that is started after the first, but must finish before the first, it might be an option to just have 2 separate transaction after one another.
    ▹ The easiest way is to just have 1 outer method which isn’t transactional which then calls 2 transactional services after one another.
    ▹ If that isn’t so easy, Springs TransactionSynchronizationManager.registerSynchronization [3] might be worth a try. That way we can register some code that runs once the current transaction finished.
  • Limiting the concurrency on the calling side to a level that guarantees the case will never be hit. This could be done in a number of ways.
    ▹ One way would be to reduce the allowed parallelism of allowed incoming connections (http).
    ▹ Another way is to introduce some other limiting-code (e.g. only around the parts of the application which is known to need nested transactions).
    ▹ The right limit depends on a few factors: the amount of nesting the application has (every level of nested calls to Propagation.REQUIRES_NEW adds another connection that is required by a single thread), the database connection pool settings and the amount of otherwise occupied db-connections.
    ▹ Also keep in mind that other parts of the application might also occupy db-connections: frequently running tasks, long-running jobs, message-queue-message-processing, RPC-endpoints etc. @Transactional(propagation = Propagation.REQUIRES_NEW) is simply a great tool which I've seen being used in problematic ways multiple times by now.

Caveats

  1. “Just increase the pool-size” is NOT a solution. And it’s not even a good workaround for any system of meaningful size — see [1]. Of course for any application the db-connection-pool should have an appropriate size — but after reading [1] we might realize that it’s lower than we initially thought.
  2. This is NOT an issue with the HikariCP Connection-Pool. If anything, HikariCP gives us great tools (like leak-detection-threshold) to identify such issues and many other parameters to properly tune our connection-pools.
  3. This is also NOT a problem/bug in Spring or even Java-specific. Any system can have this issue if it has the following property: a pooling mechanism with a fixed upper-bound and multiple nested and blocking requests to it.

Conclusion

Should we never again use @Transactional(propagation = Propagation.REQUIRES_NEW)?
No, I would not go that far.

Further Reading

[1] https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing [2] https://en.wikipedia.org/wiki/Deadlock
[3] https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/support/TransactionSynchronizationManager.html
[4] https://www.baeldung.com/java-thread-dump

Software engineer from Austria. Passionate about software, likes photography, addicted to podcasts and always busy. http://paukl.at