Clojure, specs and data correctness

My name is Edward, and I’m a senior engineer here at Funding Circle. I work in the originations side of the business, where we’re mostly concerned with taking in customer loan applications and working out whether they are eligible for lending and, if so, at what price. More specifically, I work on the decision engine. This is the component of our stack which houses our credit strategy and orchestrates the user journey.

As you might imagine, data quality and correctness matter a lot in this space, as introducing mistakes into the decision engine can be very costly. Ultimately, we are liable for any loans granted in error. Releases in the engine have a bit more friction associated with them than other components because of the cost of mistakes.

The engine is written in Clojure, a functional and dynamically typed language that we make heavy use of at Funding Circle. By its very nature, dynamic typing is more prone to runtime errors due to data mismatches than static typing. When correctness is paramount, some of the benefits that static typing brings (namely compile time errors pointing out issues with function calls and similar) can be missed.

With all that being said, there are tools for Clojure which help to alleviate these problems. Namely, clojure/spec.alpha, a tool which we use to help ensure data correctness throughout the app and help in our testing.

Specs are a tool that’s used to specify and validate the shape of data, as well as generate it. We use specs for:

• confirming that third-party data looks like we expect before using it
• checking that the outputs of the engine are valid before publishing
• providing helpful error messages when data is not valid
• implementing generative or property based tests

Specs allow the user to specify the structure of data, as well as validate it and even generate it. To illustrate, let’s take this simplified example of a map that represents a decision engine result:

{:id "df7ab223-d911-4372-a269-2adbb1564f0f"
 :status "pass"
 :offer {:interest-rate 1.02
         :amount-pence 1000000
         :term-months 24}}

A spec for this would look like so:

(s/def ::id uuid?)
(s/def ::status #{"pass", "fail", "pending"})
(s/def ::interest-rate double?)
(s/def ::amount-pence pos-int?)
(s/def ::term-months pos-int?)
(s/def ::offer (s/keys :req-un [::interest-rate
                                ::amount-pence
                                ::term-months]))
(s/def ::decision (s/keys :req-un [::id
                                   ::status]
                          :opt-un [::offer]))

Here, I’ve defined that all decisions must have an id and a status field, and optionally an offer field. I’ve then defined the requirements for each of these fields, either with a predicate function or a list of accepted values. Then, whenever we create a decision result in our code we can validate it using this spec like so:

(s/valid? ::decision my-newly-made-decision)

Using these specs, we can stop messages being published which would violate our Avro schema, by checking the message conforms before publishing. Moreover, we can use the explain function to get informative error messages that make debugging much easier:

(s/explain ::decision {:fail true})
{:fail true} - failed: (contains? % :id) spec: :user/decision
{:fail true} - failed: (contains? % :status) spec: :user/decision

This alone provides a great deal of use and is used in many of our Clojure applications, but we can take things a step further and spec functions — similar to the idea of type signatures in other languages. Consider the following function:

(defn grant-loan?
  [amount-requested term-months profit]
  (and (< amount-requested 250000)
       (> profit 1000)
       (>= term-months 12)))

The function takes three integer arguments and returns a boolean. We can write a spec for it like this:

(s/fdef grant-loan?
  :args (s/and (s/cat :amount-requested int?
                      :term-months int?
                      :profit int?)
               #(zero? (mod (:term-months %) 6)))
  :ret boolean?)

Here I’ve specified not only the types of the arguments and return values, but also that the requested loan term should be in six month intervals. These predicates are just plain Clojure, meaning the restrictions you can place on the data can be very sophisticated should the need arise. All of the above help to safeguard against run-time errors caused by mismatched data.

Lastly, we’ll look at how these specs can be used to test our code. The spec library provides functionality to take a spec and generate conforming sample data. This is useful when developing code and trying to work with data that is at least semi meaningful. Using this mechanism, we can generate decision results with valid sample data and attempt to publish them to a mock Kafka topic as part of tests that can be run locally. In this way, we can make sure that our specs are at least as strong as our schema and avoid the situation where we try and publish erroneous data.

In a similar fashion, we can also use the specs for specific functions to implement generative tests. These tests will take a specification of the inputs and outputs of a function, and verify that for random inputs that fit the input spec, we get outputs that fit the return spec. This is especially useful for catching edge cases in functions, and is a lot more effective at doing so than static values written in a regular unit test (similar to the QuickCheck library). What’s more, the kaocha plugin provides functionality that will take care of the actual testing of these functions i.e. one only needs to add the fdef code, and kaocha will generate property based tests for them using clojure.spec.test.check.

Specs are a powerful tool which help to build confidence in the correctness of your application’s data; in doing so, they can provide some of the reassurances one would get from a statically typed language. Here at Funding Circle, we make extensive use of them to test changes, validate data and help with debugging. Specs are just one of many tools and processes that we make use of to minimise risk and assure quality. If this has interested you at all, you can read more about specs here.

Thank you for reading! Make sure to follow us on Medium and our new engineering page on Linkedin.