When FP does not save us

Like many of my colleagues in the “software industry” I feel that Functional Programming has lifted my game. My first attempt at a Scala library for testing, specs, was eventually a failure because there was so much mutation under the hood that I couldn’t see a way to make this library work concurrently. On my final Java job, before starting a Scala one, I had to fix a production bug where hundred thousand dollars were sent to the wrong recipient once a month! It took me almost 3 weeks to instrument production with more log messages, read thousands of lines of code, run parts of the code with random data, to finally reproduce the issue and fix it in 2 lines of code. The bug? Unrestricted shared mutable data.

Fast-forward a few years later, armed with Functional Programming, I re-implemented my testing library which is now concurrent and based on streaming concepts; I know much better how to design immutable datatypes, control effects, and deal with shared data.

Unfortunately my sad realization is that my code is not yet bug-free and despite having a wonderful ally, the compiler, and an army of types, I am regularly losing battles. Worse, sometimes it gives me false confidence that everything looks good, only to see failures at runtime to my horror.

In this post I am going to list some of the issues which can still plague our lovely FP programs and see if any hope is in sight.

Shared data

This is an ironic situation. With functional programming we clearly try to limit the need for shared data by passing parameters with the Reader monad or changing state with the State monad. We even have very nice libraries like STM, to deal with shared mutable state and concurrency when we have to.

This does not remove the fact that we still have shared data! The prime example of this is a database. A function might still rely on the fact that some data has effectively been stored in the database to be able to properly do its job. For example we had bugs at work (in development 😅), where a function was supposed to “clean” the database between tests but was forgetting to truncate a newly-added table. Boom, failing tests. And not necessarily the easiest ones to debug because the effect is far from the cause (a classic for shared mutable data bugs).

The fix

There could be some hope with some linear type systems but honestly I wouldn’t know how to apply them to this situation. However if types don’t save us we still have tests! We wrote a “defensive” test:

1. put all the “tables to clean” in a list and use that list for cleaning
2. in the test make a list of all the tables which should be left untouched (whitelist)
3. take the list of all the existing tables in the database
4. if a table is not in the list of tables to clean but also not in the whitelist fail the test. This way we can’t forget to deal with a new table

The lesson

We rely on types a lot to model values and make sure we can only apply operations on them if they make sense. However values also have intrinsic meanings. Two lists of table names have the same type but the exact values they hold can have tremendous influence. In that case nothing is better than a test to do the checks that a human would do. A bit like double-accounting, those are boring tests: “is the value in column A the same as value in column B?”. But the day you change column B in the middle of dozens of other changes the test will have your back.

Some similar issues

To some extent you find the same situation with:

• configuration values, are we getting a secret from the right place?
• data transformation code (change the date on a reservation), does the data still make sense?
• duplicated data, do the cache and the reference agree?
• data versioning: can we still deserialize old versions of a data structure?
• APIs between systems: what do error codes and error messages really mean?

Distributed systems

So many failures fall under that category and our compilers give us very few support for most of them. There is a vast body of literature of what to expect of distributed systems. We need to be aware of the difficulties and make the right choices of databases, queuing systems, configuration software when architecting those beasts.

And at a lower level this also means that we need to:

• deal with connectivity issues by using careful retries
• make sure requests are idempotent because they can be retried (see above)
• worry about not corrupting state (this can be some of the hardest things to fix)
• know when group some actions in transactions, committing all or nothing (see above again)
• understand when it is safe to say that something is really “done”

And even if all of this is correct, eventually returns the proper values, we can still break our whole system by experiencing “perfect storms” where one failure cascades into so many others that nothing works anymore.

The fix

There are several fixes here:

• proper specifications of the systems properties with something like TLA+
• reusing heavily-tested software (FoundationDB comes to mind) and understand their guarantees
• test our own code, in the large with tools like Jepsen, and in the small with properties tests
• monitoring, to at least be warned of issues as soon as possible. Reading the logs of systems which appear to run fine can be a good idea
• spend enough time thinking of error cases and not rush features to production

Unhelpful types

To do or not to do

Many “actions” in functional programming have the same signature:

• Don’t do anything: pure () :: IO ()
• Print twice: print "hey" >> print "hey" :: IO ()
• Loop forever: forever (print "Woohoo") :: IO ()
• Fail: throwIO "oops" :: IO ()

The results of these actions are vastly different! I have had my share of head-scratching moments where the damn program was compiling but definitely not doing what I wanted to, or doing nothing, or never stopping, or…

Be more specific please

This one bit me last week. Some function parameters having the type Account . But the Account receiving the money is not the same as the one sending it! Bury this in many other parameters across several function calls where the naming is not consistent (thinkin/out , sent/received , source/target ,…) and it is very easy to substitute one account for the other.

Information loss

Every time we write

1. realResult :: Either Text a; result = toMaybe result :: Maybe a
2. realResult :: Maybe (Maybe a); result = join result :: Maybe a
3. action :: m a; result = void action

Then we deserve the hair we lose when diagnosing bugs in production. I had an example of points 1. and 2. last week (nice week…). I’m being told that my application does not display its version in a status endpoint. After looking at the code I realized that it is in fact the whole status endpoint data which is not retrieved. Butjoin was hiding that. On closer inspection it turns out that, in fact, the status was retrieved but could not be decoded and eitherToMaybe was hiding that!

I didn’t get a recent example of point 3. but definitely lost useful information like this in the past.

The fix

We are back to testing to save our bacon plus some additional linting or code reviews to detect dangerous patterns.

Conclusion: Post-Functional Programming

Mark Hibberd gave a great talk at LambdaJam 2015 on “Failure: Or the Unexpected Virtue of Functional Programming”, where he uses the term “Post-Functional Programming” to talk about other techniques for implementing reliable systems. And indeed there is so much more to learn after Typed Functional Programming: linting, testing, monitoring, specifying, distributed systems thinking, managing resources, … this is just the beginning of the journey!

Please share in the comments if you have your own favourite issues which Functional Programming does not cover, along with examples of what happened, I am sure you have some delightful war stories 😁.