Clojure: automated property-based tests for complicated inputs

Published in

AppsFlyer Engineering

12 min readAug 4, 2020

– Don’t write tests — generate them!
John Hughes (QuickCheck’s creator)

We at AppsFlyer use Clojure as the primary backend language for production. This means writing complicated business flows on a daily basis. Complicated flows can mean basically anything: business logic containing many I/O operations, complex data transformations pipelines, etc. Whatever the flow is, it requires proper testing — both unit and system tests. But the more complex the logic gets, the more complex tests we have to add. What tools does clojure offer for this purpose? A lot!

In this post, we’d like to focus on the recent and less known addition to Clojure — spec — and the broad functionality it offers. We will discuss it using a real life example¹ which lays behind AppsFlyer’s SignUp page and will include a guide to introduce clojure.spec into your project from ground up.

What is spec?

As the official guide describes, clojure.spec is a built-in library (available from clojure 1.9) which specifies the structure of data, validates or conforms it, and can generate data based on the spec. This library adds strict typing functionality into dynamically typed language, which creates broad opportunities for input validation and tests creation.

Any good API/feature/code piece starts with design. Working with sensitive user data, we in AppsFlyer have to validate our input. In case of clojure, this is where spec comes in. It is used for validating input that API/function gets. On the other hand, spec provides the ability to build automated tests. How? Let’s see in the example!

Basic flow architecture

The flow of sign up at AppsFlyer is pretty complex and includes a lot of parts, developed in different teams. It starts with the UI you can see in the URL above, then the user’s input gets into SalesForce DB. After some enrichments it is also shared with the Domain team and saved into Domain DB, which actually creates an account in our system, and eventually it is passed to some other teams. The part of the flow that we’ll zoom in is the Domain part of creating accounts.

Input

The whole flow starts with the input. It can vary and depends on the account’s business type (which are Advertiser, Agency and Partner). Basic API JSON input example²:

{ “account-type”: “Advertiser”,
  “subject”: “admin@appsflyer.com”,
  “account-data”:{
    “email”: “test-advertiser@email.ghostinspector.com”,
    “first-name”: “Test”,
    “last-name”: “Advertiser”,
    “company”: “appsflyer”,
    “website”: “http://www.appsflyer.com",
    “password”: “kjkjkklk14543”,
    “package”: “legacy”,
    <…>
  }
}

Note that here:

<…> means that it’s not the full list, production actually contains more fields which are omitted here to simplify the example;
“account-type” field’s value should be one of the predefined types mentioned above;
“subject” should be email and a user existing in the system;
“email” should be indeed email (containing “@”, some domain and a valid top level domain (TLD);
“first-name”, “last-name”, “company” and “password” should be non-empty strings;
“website” should be either nil or a valid website — string containing “http://” or ”https://” and a valid TLD;
“package” should be either nil or non-empty string with the business package name existing in the system;
The field set provided in the example is relevant for this account type only; additional fields are expected for other account types as must-have.

To validate it, we can use regular clojure.spec built-in functions like:

(ns my-project.validator
  (:require [clojure.spec.alpha :as s]))
  (s/def ::password (s/and string? not-empty))

But it’s pretty clear that it turns to be not enough once we have more complex types like, e.g., email. For this case we’ll need something like:

(def email-regex #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")
(s/def ::email-type (s/and string? #(re-matches email-regex %)))
(s/def ::email ::email-type)

When creating a new user, we also need to check that the user’s email does not exist in our DB yet. So it’s obvious to add this check into validating function as well using spec and some additional function that goes to the DB and searches the provided email in it³:

(defn- existing-user?
  "Checks that email provided exists in DB"
  [email]
  (does-user-exist? email)) ;placeholder for your DB search here ;)

So, combining two conditions described above, we’ll get:

(s/def ::email (s/and ::email-type #(not (existing-user? %)))) ;validates that email is of a specific type and  user does NOT exist ;in DB

We can combine all those small specs into full input spec ::create-account-input (using keys), as well as validating functions into a general validator:

(defn create-account-input-valid?
 [input]
 (if (s/valid? ::create-account-input input)
  {:valid? true}
  {:valid?  false
  :reason  (str "Invalid input format: " (s/explain-str ::create-account-input input))}))

So having a validator like that, we are getting into the situation of using not only built-in clojure functionality, but self-written functions as well. Which is obviously a possible place for bugs. How to make sure we haven’t created any? Adding tests, of course!⁴

Testing & generators

Obviously, we need to test our validator as a black box with unit tests. Unit tests paradigm expects testing all the fields’ possible combinations — valid and invalid — and checking that the output meets expectations.

But having that big input and so many validation rules creates an endless quantity of fields’ values permutations. Under those circumstances, it’s not possible to write tests for all the edge cases and permutations manually. What are we to do then? Now it’s time to use the power of spec, which will allow us to:

Generate the needed input for tests in an automated way
Build automated property tests with no manual work.

Having generators inside it, spec can create all those permutations for us. As we know from the documentation, built-in spec generators can generate anything conforming standard clojure types:

(ns my-project.validator 
 (:require [clojure.spec.alpha :as s]
   [clojure.spec.gen.alpha :as gen]))(gen/generate (s/def ::password (s/and string? not-empty))) ;will generate non-empty string

But as we saw in our input example, it is not enough in real life, when it comes to cases like checking that the “email” value we get is actually an email. Then custom generators come to help. The idea is pretty straightforward — as clojure itself cannot generate emails conforming our spec out of the box, we need to tell it how to generate the required value or even give it a set of sample values. Indeed, let’s explore an example of giving rules to generate email⁵:

(def email-regex #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")(defn- email-generator
"Generates email with string@string.co.il for tests auto-generated input"
 []
 (gen/fmap #(str % "@" % ".co.il") (gen/string-alphanumeric)))(s/def ::email-type (s/and string? #(re-matches email-regex %)))(s/def ::email (s/with-gen ::email-type email-generator)) ;validates that user does NOT exist in DB

As we can see from the code above, email-generator function is a generator that knows how to create strings according to our “email” rules. Works great for this example.

But what if we need our email to exist in the DB like the “subject” field? The code we’ve just explored has no idea if the email exists in some external resource. To explain to clojure how to do it as well, we can add a part that takes generated email and checks if it exists in the DB into email-generator, but it will need to sample so many emails before it actually finds one existing in the DB, which is not efficient anymore. In this case we can use another option generators provide us — giving it a set of ready-made examples meeting our validation rules:

(s/def ::subject (s/with-gen ::email-type #(s/gen #{"test-0@appsflyer.com" "test-1@appsflyer.com"})))

So this way we can add generators to any input field we have. This will let us generate all the fields we need just calling the spec from our validator namespace and `generate` function from spec library:

(ns my-project.unit-tests 
 (:require [my-project.validator :as validator]
   [clojure.spec.gen.alpha :as gen]))(defn generate-input-template 
  "Generates hash-map account input by a predefined template"
  [] 
  {:account-type (gen/generate (s/gen ::validator/account-type)),
   :subject      (gen/generate (s/gen ::validator/subject)),
   :account-data
   {:package     (gen/generate (s/gen ::validator/package)),
    :email       (gen/generate (s/gen ::validator/email)),
    :salt        (gen/generate (s/gen ::validator/salt))
    :website     (gen/generate (s/gen ::validator/website)),
    :company     (gen/generate (s/gen ::validator/company))}})

Looks good, right?

But not good enough: we would expect something like one-line generation and test like seen in clojure.spec test documentation examples:

(require '[clojure.spec.test.alpha :as stest])(s/gen ::validator/create-account-input) ; generating input of the ;same structure as a JSON at the beginning of the post
(stest/check `validator/validate-account-input) ; unit test checking ;validator function

But how can we achieve it, having nested fields in the input structure (see “account-data” nested map in the input example)? Moreover, as we’ve mentioned above, the JSON input example at the beginning of the post is just a basic example and will have additional must-have fields for other account types, e.g.:

{"account-type": "Partner”,
   "subject": "admin@appsflyer.com",
   "account-data":{
    “partner-acc-id”: “testPartner0”,
    "email": "test-partner@email.ghostinspector.com",
    "first-name": "Test",
    "last-name": "Partner",
    "company": "appsflyer",
    "website": "http://www.appsflyer.com",
    "password": "996jklhef",
    "package": "legacy",
    <...>
   }
}

The previously discussed fields can be found here, as well as the “partner-acc-id” field which is a must for Partner accounts (and must be a non-empty string), though it must not exist for other account types. Thus, there is a dependency between fields: existence of the “partner-acc-id” field depends on the “account-type” field’s value.

So it’s easy to see that this structure cannot be generated by clojure.spec automatically. Don’t worry — clojure.spec is full of surprises and has power to cope with this challenge as well!

Multi-spec power

Clojure.spec provides us multi-spec tooling to create different sets for different types of the same input. Based on that, we can add a selector indicating the type of input (account type in our case), a base set of fields relevant for every account type and a separate method with its own additional fields for each account type required:

;account type selector which is actually the same to account type ;possible values
(s/def ::account-type-key (s/with-gen keyword? #(s/gen #{:Advertiser :Agency :Partner}))) 
;case switcher for multispec with the values similar to “account-type” values(s/def ::base-input 
       (s/keys :req-un [::email ::password ::first-name ::last-name ::company]
               :opt-un [::website ::package]));multi-spec is used to identify whether the instance type needs ;special props 
;e.g., partner-acc-id for Partner accounts only(defmulti account :account-type-key)
(defmethod account :Advertiser [_]
   (s/merge (s/keys :req-un [::account-type-key]) ::base-input))
(defmethod account :Partner [_]
   (s/merge (s/keys :req-un [::account-type-key ::partner-acc-id]) ::base-input))
(defmethod account :Agency [_]
   (s/merge (s/keys :req-un [::account-type-key ::agency-field]) ::base-input))
(defmethod account :default [_]
    (s/merge (s/keys :req-un [::account-type-key]) ::base-input));creating nested account-data structure
(s/def ::account-data (s/multi-spec account :account-type-key));creating the whole input structure and defining fields dependency
(s/def ::create-account-input 
  (s/and (s/keys :req-un [::account-type ::subject ::account-data])
    (fn [{:keys [account-type account-data]}]
      (= (:account-type-key account-data) (keyword account-type)))))

This piece of code gives us an ability to generate maps of the same structure as the JSON we have as an input (and which is eventually also parsed to a map), including one additional field of :account-type-key which was used as a selector. Its presence does not affect unit tests, but can be dissoced if needed. Let’s check out the output:

(gen/generate (s/gen ::create-account-input))

will create the following example outputs: they differ both in account types/partner-acc-id presence and other fields presence (Partner output is missing :package and :website, and it’s okay since they can be nil, according to our spec/validation rules)⁵:

{:account-type "Advertiser",        {:account-type "Partner",
 :subject "test-0@appsflyer.com",   :subject "test-1@appsflyer.com",
 :account-data                      :account-data
  {:account-type-key :Advertiser,    {:partner-acc-id "P6Jcwp1FKm3",
   :package "legacy",                 :email "zP8zAx@zP8zAx.co.il",
   :website "http://www.9EWsC.com",   :password "Of6wP55QD",
   :email "yclXhm2@yclXhm2.co.il",    :first-name "26rjol",
   :password "0s2TZv5",               :last-name "Nht9yP4pdv69",
   :first-name "JCN69Ya85Klx",        :company "YcOFdLRSkt",
   :last-name "54e37ezGUII9UeTX",     :account-type-key :Partner
   :company "7f29khk"                 <...>
   <...>                             }}
}}

Now we have got a one-line complex input generation — the first goal achieved!

What now? Now it is time to create one-line unit tests and explore clojure.spec generative opportunities for tests in general!

Tests

We can be sure now that clojure.spec itself can generate our complex input in every possible permutation automatically. It means that now clojure does not need us to do manual work to test the validator function, stest/check should be enough.

Let’s assume that we would like to test the mentioned earlier create-account-inout-valid? validator function. According to the documentation, to be able to do it in an automated way, we need first to add fdef to define specifications for our function:

(s/fdef create-account-input-valid?
  :args (s/cat :input ::create-account-input))

Now we basically can call

(ns my-project.unit-tests 
 (:require [my-project.validator :as validator]
   [clojure.spec.test.alpha :as stest]))
 
 (stest/check `validator/create-account-input-valid?)

It will only return

;;=>({:spec #object[clojure.spec.alpha$fspec_impl$reify__13728 ...],
;;  :clojure.spec.test.check/ret {:result true, :num-tests 1000, :seed 1466805740290},
;;  :sym spec.examples.guide/ranged-rand,
;;  :result true})

which is not that informative. To get some human readable results we can add the following wrapper:

(deftest test-create-account-input-valid-spec
  (testing "Testing input conforming spec"
    (refresh)
      (timbre/set-level! :report) 
      (let [results (stest/check `validator/create-account-input-valid?)]
        (run! #(timbre/report %) results)
        (is (every? nil? (mapv :failure results))))))

In case of success it returns:

REPORT [create-account.create-account-unit-test:37] - {:spec #object[clojure.spec.alpha$fspec_impl$reify__2524 0x34934bc8 "clojure.spec.alpha$fspec_impl$reify__2524@34934bc8"], :clojure.spec.test.check/ret {:result true, :pass? true, :num-tests 1000, :time-elapsed-ms 53011, :seed 1585735274690}, :sym my-project.validators.validator/create-account-input-valid?}

In case of error, it will display the relevant exception.

Basically, this is it — our second (and the main) goal is achieved!

But if we have passed such a long way understanding clojure.spec, let’s see what other abilities it gives us.

The other interesting opportunity is to use the power of clojure’s test.check library combined with spec’s generators, which offers a broad variety of property-based test tools. We’ll check out only the basic one here:

(require ‘[org.clojure/test.check "0.10.0"])(def positive-answer {:valid? true})(def validator-property-test
  ;generating input
  (prop/for-all [input (s/gen ::validator/create-account-input)] 
    ;how to get the result
    (let [result (validator/create-account-input-valid? input)] 
      ;success condition
      (is (= positive-answer result)))))(deftest input-validator-unit
  (testing "Creating new account: input validator unit tests:"
    (testing "Validator property testing" (tc/quick-check 100 validator-property-test)))) ; test itself

The nice option here is having control of basically everything: input, operation to check, success criteria and number of tests to run. This function expects more optional arguments like max-size, seed, etc.

Another option is combining spec’s generators, test.check and manual checks:

(def positive-answer {:valid? true})
(def negative-answer {:invalidity-type :invalid-permission
                      :reason "Subject does not exist or is not an existing user"
                      :valid? false});input validation unit tests
(deftest input-validator-unit
  (testing "Creating new account: input validator unit tests:"
    (testing "Validator property testing" (tc/quick-check 100 validator-property-test))
       (testing "Positive"
         (is 
           (= positive-answer 
              (validator/create-account-input-valid? 
                 (gen/generate (s/gen ::validator/create-account-input))))))
       
       (testing "Negative – subject does not exist"
         (is 
           (= negative-answer 
           (validator/create-account-input-valid? 
               (assoc (gen/generate (s/gen ::validator/create-account-input)) :subject "some-random-email-that-does-not-exist@appsflyer.com")))))))

Thus we can manually inject some field instead of the generated one. This approach can be used if we’d like to check some specific scenarios or when having manual validations in addition to specs.

Furthermore, clojure.spec/generators can be used to generate input for system tests.

Conclusion

Clojure.spec is still in alpha and tends to look complicated at the beginning, but it offers a broad functionality that can simplify the developer’s life. You might think of using it developing APIs/functions, where:

Input types do matter;
Having complex structured input;
Having a large number of input fields;
Property testing is needed, etc.

If you’d like to introduce automated property-based tests into your clojure project, just follow the following steps:

Add clojure.spec dependencies;
Add specs for your input fields;
Create general validator function containing specs and your custom validations;
Add custom generators into your specs definitions where applicable;
Add `fdef` definition to your validator function;
Use stest/check, stest/check or their combination;
Enjoy!

Thanks to Nir Rubinstein, Morri Feldman and Ben Sless.

[1] Omitting some technical details to reduce the example’s complexity (e.g., password encryption, etc.).

[2] This is a very simplified example that doesn’t fully reflect the real production input and validation process that we do, but it’s sufficient for our example.

[3] Mind that this function omits the actual process of searching the DB to simplify the example.

[4] Note that tests cannot guarantee that there are no bugs at all for 100%.

[5] Remember that this is a very simplified example that doesn’t fully reflect the real production input, validators and generators, but it’s sufficient for our example.