Transactional REQUIRES_NEW considered harmful — Spring/Java transaction handling pitfalls

The Issue

The Spring @Transactional Annotation is a powerful tool to make transaction-management easy for developers.

What is Propagation.REQUIRES_NEW?

Often people think of this propagation setting as “magically tells the database to make a nested transaction”. However this mental model is wrong — many Databases don’t even have nested transactions, so something else needs to happen.

Transactional and Transactional(REQUIRES_NEW) interacting with the Connection-Pool. “Connection 2” is just an example of any currently idle connection.

Deadlocks on all Sides

Quick recap on deadlocks: “[…] a state in which each member of a group waits for another member, including itself, to take action […]” from [2]

  • There 2 (or more) threads each have acquired a database-connection (and thereby emptied the java db-connection pool).
  • Now each thread wants to acquire another connection and thereby waits for the other threads to release them.
  • The other threads to the same, so they would only release a connection after they got an additional one.
  • The Java-Process has opened a connection and e.g. inserted a new user in a table.
  • Then the Java-Process (same thread) has opened another connection (via a call to a @Transactional(propagation = Propagation.REQUIRES_NEW) annotated method)
  • In this second connection, it tries to insert something in a user-preferences table and use a foreign-key to the just inserted user.
  • When the inner method is left and Spring tries to flush the “inner”-transaction (second-transaction is more accurate), the database will block.
  • The database waits for the first connection to complete (commit) before it lets the second one commit.
  • This is completely valid from the database’s point of view, since those 2 connections are completely independent.
  • However the Java-side introduced a blocking dependency which runs in the opposite direction (the inner method won’t finish/return, so the calling method of course can’t finish either).

Reproduction

If you want to code/play/… along, here is the example repository with the code.

seq 1 8 | xargs -I $ -n1 -P10  curl "http://localhost:8080/nested"

Identifying The Issue

What can we do if we think our application has this problem? It may just be a feeling, but how can we be sure?

Shortened Stacktrace showing 2 threads stuck at datasource.getConnection()
Pool-status showing us that lots of connections are waiting (requested from the pool, but could not be provided yet).

Possible Solutions

There are basically 3 ways this can be fixed:

  • Getting rid of the “nested” transaction. It might not be necessary at all to have 2 separate transactions. If that’s the case, just get rid of it by using the default propagation level.
  • Serializing the 2 transactions. Instead of having an inner transaction that is started after the first, but must finish before the first, it might be an option to just have 2 separate transaction after one another.
    ▹ The easiest way is to just have 1 outer method which isn’t transactional which then calls 2 transactional services after one another.
    ▹ If that isn’t so easy, Springs TransactionSynchronizationManager.registerSynchronization [3] might be worth a try. That way we can register some code that runs once the current transaction finished.
  • Limiting the concurrency on the calling side to a level that guarantees the case will never be hit. This could be done in a number of ways.
    ▹ One way would be to reduce the allowed parallelism of allowed incoming connections (http).
    ▹ Another way is to introduce some other limiting-code (e.g. only around the parts of the application which is known to need nested transactions).
    ▹ The right limit depends on a few factors: the amount of nesting the application has (every level of nested calls to Propagation.REQUIRES_NEW adds another connection that is required by a single thread), the database connection pool settings and the amount of otherwise occupied db-connections.
    ▹ Also keep in mind that other parts of the application might also occupy db-connections: frequently running tasks, long-running jobs, message-queue-message-processing, RPC-endpoints etc. @Transactional(propagation = Propagation.REQUIRES_NEW) is simply a great tool which I've seen being used in problematic ways multiple times by now.

Caveats

  1. “Just increase the pool-size” is NOT a solution. And it’s not even a good workaround for any system of meaningful size — see [1]. Of course for any application the db-connection-pool should have an appropriate size — but after reading [1] we might realize that it’s lower than we initially thought.
  2. This is NOT an issue with the HikariCP Connection-Pool. If anything, HikariCP gives us great tools (like leak-detection-threshold) to identify such issues and many other parameters to properly tune our connection-pools.
  3. This is also NOT a problem/bug in Spring or even Java-specific. Any system can have this issue if it has the following property: a pooling mechanism with a fixed upper-bound and multiple nested and blocking requests to it.

Conclusion

Should we never again use @Transactional(propagation = Propagation.REQUIRES_NEW)?
No, I would not go that far.

Further Reading

[1] https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing [2] https://en.wikipedia.org/wiki/Deadlock
[3] https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/support/TransactionSynchronizationManager.html
[4] https://www.baeldung.com/java-thread-dump

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Paul Klingelhuber

Paul Klingelhuber

80 Followers

Software engineer from Austria. Passionate about software, likes photography, addicted to podcasts and always busy. http://paukl.at