…properly

Exop on StreamData

Hello elixir developers!

I’ve been absent here on medium for a while: too much to accomplish + have been working on a new library which I started to use in my elixir projects in production in order to test it.

Today I’d like to tell you a story how I was inspired by Andrea Leopardi’s talk “Property-based testing is a mindset” he gave at ElixirConfEU 2018 in Warsaw, Poland, and what is ExopData as the result of this inspiration.

You’ll find out what are Exop and StreamData with a brief description of these libraries. Next, I’m going to provide you the idea behind ExopData, how it was started and evolved, which tricky moments we’ve faced during the implementation. And how ExopData can help you with data generating or property-based testing.


Exop

Earlier (actually, almost 2 years ago) I’ve presented Exop — the library which significantly helps you to organize your project’s code in more DDD-way, by splitting your code in dedicated modules. One module — one business operation.

Exop also deals with incoming parameters validation and eliminates boilerplate code, so you need to deal only with what really matters — business logic.

Apart from validation and unified (an operation returns either {:ok, _} or {:error, _} tuple), Exop offers you:

  • interruption feature (early return)
  • parameters coercion mechanism
  • operation’s policy check (authorization)
  • fallback flow
  • …and much more

I’ve been using this library in almost every project and work with Exop’s operation on daily basis (this is why this library constantly been improved).

With Exop I wanted an operation (simply a module) has some public contract (described by a few parameters). Which is easy to find (at the top of an operation’s module) and easy to read.

Simply, take a look at a contract: and you understand what you need to successfully run an operation. Try it out! Exop worth it.

very simple Exop operation

StreamData

title slide of Andrea Leopardi’s presentation at ElixirConfEU 2018

I was sitting in the 4th row in the hall when Andrea started his talk about property-based testing. Please, check it out on YouTube — it is very cool.

During this talk, Andrea presented StreamData library: a library which provides a set of data generators. You can use it to generate data in your project (obviously) and write property-based tests.

examples from StreamData README

I don’t want to describe how cool StreamData is, which features it has and which possibilities it offers to Elixir developers — feel free to find it in Andrea’s presentation.

One particular thing was interested me a lot during the talk. Andrea showed a few slides and was telling about the possible future and future features of StreamData:

it is possible to utilize Elixir specs and generate data by specs

And I (as Exop author) at that moment was like:

wait a second…

Take a look at Exop operation:

particularly, at this part:

💡 Eureka!

That what Andrea mentioned as StreamData’s possible feature could be done with Exop, and not for a certain function, but for your business logic module. Because, Elixir specs are contracts themselves, and I in my Exop’s operations already have contracts, and all the information I need to generate data with StreamData by utilizing it’s predefined and custom generators.

This is why returned from the conference in the next few months I started to implement…


ExopData

The goal of this library is to help you to write property-based tests by utilizing the power of Exop (and its contracts) and StreamData.

ExopData shares the same contract principle with Exop. In order to generate data you need to define a contract (a set of parameters), contract description rules are derived from Exop.

For all possible checks (opts) you can refer to Exop documentation.

Here are examples of how to generate data with ExopData:

using a dedicated contract (described with a list of maps)
generate for an operation’s contract (described with a set of parameter/2 calls)

As you can see: having your logic organized with Exop operations starts to give an additional benefit — data generation with ease.

ExopData is good enough to cover your needs in really complex data structures:

crazy structures are welcome

So, we can generate data for operations contracts now let’s return to the tastiest part — property-based testing.


I have not set a goal to reinvent StreamData’s property-based testing approach, didn’t want to hide everything behind a DSL — just provide a convenient way to generate complex data with StreamData.

generate test data for a contract defined in tests

In the example above you don’t need to have Exop, this code could be easily rewritten with StreamData. Because ExopData.generate/2 returns StreamData generator (we will talk about the implementation a bit later).

ExopData starts to shine when you use it with Exop operations. Why? Because property-based testing in this case even easier, super-easy:

generate test data for an operation

You already have MultiplyService somewhere in your project (or you will have). ExopData can get all necessary information only by your operation’s module name and start to generate test data.


Moreover, ExopData with Exop operations makes TDD as it should be.

What is the most challenging part in TDD? In my opinion: to understand, know in advance your project’s unit (a module in our case) public interface (public functions + their arity). The first red-green-refactor iteration differs from the last one a lot. Test suits have to be updated as many times as you change your module’s public interface.

You can forget about it with Exop + ExopData: Exop unifies an operation input and output (input: map or keyword, output: tuple) and ExopData takes just the operation’s module name (an atom), searches for a contract and generate data depending on what you defined in this contract inside your operation.

In the example above: no matter how you will change your business logic inside MultiplyService.process/1 function, the test will remain 100% the same (until you change the output value, so there will be a need to change expected result…only).

So it is possible to start from property test:

Which fails because there is no MultiplyService, than because module is not an Exop.Operation, than because it doesn’t define any parameter, etc. And finally, iteratively we will get a valid operation:

Once again: ExopData is not a replacement for StreamData, it just offers you a handier (in our, authors, maintainers opinion) way to use it, define custom generators, generate complex data structures.

Key features

  • it offers a convenient way to describe and generate complex data structures (with ‘list_item’ and ‘inner’ options)
  • it allows you to use your own custom generators (based on StreamData generators) or exact values
  • it simplifies test data generation for Exop operations
custom generator (with complex nested structure) provided to ExopData.generate/2
how to use a certain value (if you want to stub some data), even in nested contract as well

What is under the hood?

StreamData offers a nice approach for custom generators creation by combining simple predefined generators:

an example from https://github.com/whatyouhide/stream_data README

So, the first attempt was to assemble gen all macro + data <- generator clauses as AST within a macro. That was challenging, hard to read, maintain and add new features but worked somehow. It looked like this:

gen clauses were assembled one by one with bare-metal AST
literally I collected AST piece by piece manually…

Later, my friend and another ExopData maintainer Aleksandr (llxff) took a look at this mess and asked:

Hey, why is it so complicated? All you need is to generate parameters for Exop operation. Parameters could be either map or keyword. So, why don’t you use StreamData map generator?

That was the great question, I felt stupid. That was so simple and obvious, why didn’t I think about it?

He refactored this ugly AST part and from that moment it became much easier to add new features.

Basically, ExopData under the hood is a custom map generator each clause of which is StreamData generator with some options provided. Here is how simple generator for parameter(:param_a, type: :integer, numericality: %{min: 1, max: 10}) is constructed and looks like:

briefly how it works now
an example of generator module for :string type

More complex options passed to ExopData like list_item or inner work in the same way but construct map generator recursively.

Limitations and challenges

Every property test should generate a lot of data and do it fast. This is why we had to add some limitations to ExopData. Those limitations mainly were brought for really complex data structures generating, but there are some cases when we have to keep performance <> data quality balance. However, some of these limitations might help you to think about your operations incoming parameters.

Let’s say you have an operation which takes some structure or map. It is totally ok. But if this structure/map has a nested structure with a few levels it might take a lot of time to generate hundreds of such structures/maps in your property-based test. In this case, consider making your operation’s contract (parameters) flat. Most probably you’re not going to use all values from the whole structure/map.

Consider this example:

how CheckRole operation might look like with different contracts

In the example above your intent is to check a user role. A user has been placed by some plug into a connection assigns as ‘user’.

The first approach: you pass to CheckRole operation the whole Plug.Conn structure which contains 28 keys. And you’re actually interested in the only one — assigns, within its map value you’re interested only in user key’s value — role. In this case, ExopData will have to generate the whole complex structure of Plug.Conn, it will take some time. Maybe it will be fast (depends on your hardware), but obviously it will be much slower than generate just a string for role parameter for the second approach CheckRole operation.

If you’re not happy with the smallest granularity (of a parameter) consider the third CheckRole operation module: pass User structure, validate its role parameter. It will be much efficient to generate just User structure.

And it is much easier to figure out what is going on here CheckRole.run(role: conn.assigns.user.role) or here CheckRole.run(user: conn.assigns.user) than here CheckRole.run(conn: conn). CheckRole.run(conn: conn) #which role we’re going to check here?

Since I’m not just the author of Exop and ExopData maintainer, but an active user of those libraries, I’ve revised a number of operations modules in my project as well. They became cleaner and this made tests much more efficient.

All known limitations we described in the limitations part of ExopData’s README. Please, check it out.

The next thing which worries us: StreamData library status. A few commits for the last six months. That is sad and if its maintainers decide to stop working on it will impact ExopData of course.

Nevertheless, we have plans for the future :)

Future plans

  • provide a wrapper for check all params <- ExopData.generate(), do: …
where the last statement will be asserted with MultiplyService.run/1 result
  • generate commands for an Exop operation (stateful testing)
  • consider PropEr as possible generator
  • …and of course constant improvements based on real usage feedback

If you find either Exop or ExopData interesting — please try it out, share your experience, submit issues and PRs. Clap, star, share, submit.


Many thanks to Aleksandr Fomin (llxff) for ideas and implementations, Andrea Leopardi (whatyouhide) along with Elixir core team for the inspiration and their work, and you Exop/ExopData libraries users for using them :)