Stateful Components in Clojure: Part 1
You’re building a web app with Clojure, and things are going great. Your codebase is full of these fantastic pure and composable functions, you’re enjoying all the great ways in the Clojure ecosystem to manipulate HTML and CSS, generate data for your web APIs, and maybe you’ve even written a Clojurescript front-end. It’s all smooth sailing until you realize that it’s time to connect to a database…
If you’re like many Clojure developers, you’ve lost some time learning about various bad ways to solve the problem of managing stateful things like database connections, caches, queues, authentication backends, and so on in Clojure apps.
Let’s look at the hypothetical journey of one developer — we’ll call him Dave, since everyone knows a Dave — and his attempts to solve this problem.
Dave’s Journey
Dave is building a web app. Probably one that’s a lot like yours. And he too is just now running into the problem with stateful services, as he needs to add a database to his app.
Dave is first tempted to avoid the state problem entirely — a noble impulse for any engineer — by stashing the database config in a dynamic var, reasoning with some justification that it’s just data so that’s not so bad. He ends up with the following:
(def ^:dynamic *db-config* "postgresql://localhost/devdb?user=dev&password=dev")(defn select
[query]
(jdbc/query *db-config* query))
This approach certainly has its benefits. None of Dave’s handlers have to know or pass around anything about the database; they simply call the select function. Dave is initially satisfied with this approach. It looks pretty easy, so he writes some tests.
(deftest homepage-handler-test
(with-bindings {app.db/*db-config* test-config}
(is (re-find #"Hits: [0-9]." (homepage-handler {})))))It’s reasonably testable code, so long as he always remembers to correctly bind the *db-config* var. He keeps working on his code until he notices that some of his pages seem to be loading slowly, especially the ones that do a lot of database queries. Pretty soon the reason becomes clear — Dave’s app has to open a new connection to the database for every query.
The obvious solution is to use a persistent database connection. The question is where to put it? Using a dynamic var to hold the config is one thing — that’s just data — but a database connection is a stateful object. Like so many have done before him, Dave accepts a compromise on his functional programming principles and decides to stash the database connection in a delay. Now his code looks like this:
(def ^:dynamic *db-config* "postgresql://localhost/devdb?user=dev&password=dev")(defonce db-conn (delay (make-pool *db-config*)))(defn db []
@db-conn)(defn select
[query]
(jdbc/query (db) query))
Testing is now a bit tricker. He has to redef the db function and handle setup and teardown.
(def test-db-conn (atom nil))(defn with-test-db-conn
[f]
(reset! test-db-conn (make-pool test-config))
(f)
(shutdown @test-db-conn)
(reset! test-db-conn nil))(use-fixtures :once with-test-db-conn)(defn test-db []
@test-db-conn)(deftest homepage-handler-test
(with-redefs [db test-db]
(is (re-find #"Hits: [0-9]." (homepage-handler {})))))
Dave is feeling nervous. It’s a nagging, itching sort of feeling in the back of his mind, like the first stages of athlete’s foot. This code is starting to bother him. Just this little bit of global state is already requiring a lot of machinery to work with. His worries haven’t erupted into a full scale breakout yet, however, and he’s got more story points to complete.
His next task is to add a new feature that’s so important that his boss’s boss’s boss is regularly asking for status updates from his boss’s boss, which get translated into urgent requests to his boss, which finally become exhortations so anxiety-ridden that they are nearly incoherent to him. Their web app is the premiere service for posting pictures of cats wearing socks, and Dave needs to add a feature that either watermarks the pictures of these unfortunate animals with the site’s URL for non-paying users, or with a user-selected message for paying users.
To do this he needs to build a system that reads from a job queue in the app’s database and processes each new image to add the appropriate message. He’ll need to have several workers doing this for performance reasons, since few things have less patience than a cat forced to wear socks. This is a tricky problem for his approach to state management because he needs to be able to initialize and shut down these workers. Not only that, but they depend on the database connection.
Here’s what he comes up with.
(def ^:dynamic default-message "Posted to www.mssngvwls.io")
(def ^:dynamic number-workers 3)(defonce stop? (atom false))(defn start-workers []
(reset! stop? false)
(dotimes [worker-n number-workers]
(future
(while (not @stop?)
(if-let [task (select-next-task (db))]
(do
(add-watermark (:image task) (or (:message task) default-message))
(complete-task (db) task))
(Thread/sleep 500))))))(defn stop-workers []
(reset! stop? true))
He adds a call to start-workers to his main function, and it seems to work well enough. He wasn’t able to do much iteration at the REPL while developing because he kept needing to manually restart things, and it was tedious to keep things pointed at the right database. His test code needs to redef even more things now. There’s still the database redefs from before along with the custom setup and teardown testing code, and now he also has to redef the stop? atom so that he can test this worker code that uses a completely different start and stop mechanism.
Dave’s morale is down, too. Nothing about this code seems functional anymore — he may as well have done it in Java. It just feels wrong to him, but he doesn’t see a better way out. He’s reached the low point in his journey. What Dave needs is a hero, and one arrives in the unlikely form of Stuart Sierra.
posted from Ladders Engineering Blog
