5 lessons learned from my continuing awesome journey with ZIO

Published in

Wix Engineering

6 min readSep 13, 2020

Back In January 2020, I wrote two posts (I, II) about pitfalls to avoid when starting to work with ZIO.
9 months have passed. In the interim period, ZIO 1.0 was released and ZIO Environment has been tremendously improved with the introduction of ZLayer.

From my experience it is a real joy to write code with ZIO, especially in cases where you have complex concurrent/async requirements which are much simpler to implement with the features this library provides.

Since January, I’ve written a lot more code with ZIO and had a few more mistakes I’ve made and corrected along the way. Below I gathered 5 more ZIO lessons learned. Lessons ranging from the best way to write recursions with ZIO, to the correct way to write test assertions when using TestClock.

1. Performing Non heap-safe Recursion

Recursion is used widely in functional programming.
Non tail recursive functions (where the call to the recursive function itself is found at the end of the function) risk causing a memory leak and even stack overflow errors.

The ZIO interpreter makes sure to keep recursions stack safe by switching call frames from the JVM stack to a heap allocated stack. No more stack overflow errors, but still potential memory leaks.

Consider the following simplified code example of our implementation of Kafka consumer message polling and handling.

Our first attempt included a recursive call (pollLoop) that allowed for continuous polling efforts. But even though the last call is to the recursive function itself, this implementation was still not heap safe, and indeed on stress tests we noticed a memory leak of hundreds of MBs in a short period of time.

The problem is that we used a for comprehension, which is implemented in a way that does not allow tail recursion. A for comprehension is basically a sequence of flatMaps followed by a final map. The code below shows the de-sugarized version of the for comprehension above. Notice that because of the identity map function, the last call is not to the recursive function.

In our Greyhound codebase, we decided to stop using recursive methods completely and instead use the ZIO operator doWhile that is guaranteed to provide a heap safe tail recursion of the effect it operates on. For the case above, we changed the recursive method to just perform a single poll operation (pollOnce) and the doWhile operator manages the recursion:

pollOnce(running, consumer).doWhile(_ == true).forkDaemon

So the pollOnce implementation should return a UIO[boolean] stating whether to continue recursion or not:

And indeed the subsequent stress tests showed the memory leak vanished.

2. Running a side-effect alongside an effect that is repeated

Mixing lazy and eager code inside a method is always problematic (see lesson 2 in part 2), especially when the intention is to repeat its effect. Consider the following code example:

Any non-ZIO side-effect that is executed in publishQuote method will not be repeated by the repeat operator found in repeatedlyPublishQuote. The repeat operator only repeats functional effects.

Functional effects are composed values that the ZIO runtime can actually interpret and execute. This is not the situation with side-effects that usually perform some IO operation and don’t provide any information for the runtime to work with.

In our example if you want to see continuous log entries for the repeat effect, use a functional effect such as console.putStrLn and put it inside the for-comprehension like the following example:

Also, if you want to make sure the effect is repeated, add a catchAll operator after the for comprehension to make sure it never fails, otherwise it will stop repeating on the first failure.

3. Unintentionally using TestClock in periodic assertion code

Let’s say we want to test a feature called “delayed message consumer”. e.g. a consumer that only processes messages after a certain configured delay.
Our test will include a consumer configured with 1 second delay and a producer that produces the message.

The fake messageHandler is provided with a counter that allows the test to assert that the message processing has indeed happened. Once the message is produced, we check the counter periodically until it satisfies the predicate.

Note: this example is written with specs2 but the same principle applies to ScalaTest

In order to reduce run time of the test, we can use the great TestClock feature that ZIO has, in order to artificially change the time to be 1 second in the future. But we should use TestClock with great care and beware of unintended consequences.

Let’s look at the implementation of eventuallyZ:

By using a ZIO Schedule, it invokes the provided predicate every 100 milliseconds until the predicate returns true or a timeout occurs after 4 seconds.

But there’s a problem here. eventuallyZ expects a Clock in the environment, but it does not specify which Clock. It should be using the Live Clock, but in our example, because the test uses TestClock, then eventuallyZ is also going to use the TestClock, which means no actual periodic Scheduling will occur as it does not call TestClock.adjust.

One solution to this issue is to simply provide the correct clock using provideSomeLayer ZIO operator:

Now ZIO effects running in eventuallyZ scope use the Live Clock and correctly run the predicates every 100 milliseconds. This does not affect the rest of the test code which can still utilize TestClock.

For a complete code snippet of this example go here.

4. Forgetting to map between ZIO Test assertions

While we are on the subject of tests, I’ve also had the opportunity to experience working with the wonderful ZIO Test library. Here is a basic example of how to test if a number is positive and even:

You can see how easy it is to execute the test on a generated stream of values, which can of course be a stream of random values for property based testing (e.g. positive integers: Gen.anyInt.filter(_ > 0)). But there is a small issue in the code above. Only the “isEven” assertion will actually be executed, as the first assertion is not mapped to the second one, so this test will pass even though 0 is not a positive number.

+ positive and even after addition
Ran 1 test in 660 ms: 1 succeeded, 0 ignored, 0 failed

In order to fix this all that is needed is to map the two assertions using the && operator:

assert(number)(isPositive) && 
assert(number % 2 == 0 (Assertion.isTrue)

Now the test fails:

Ran 1 test in 685 ms: 0 succeeded, 0 ignored, 1 failed
- positive and even after addition
Test failed after 1 iteration with input: 0
0 did not satisfy isGreaterThan(0)

5. Interrupting a fiber that originates from a Managed#Acquire scope

ZIO fibers are the building blocks for all the Concurrent / Async features that ZIO provides.

Most of the time we use built-in operators like foreachPar that create fibers for us.

Sometimes we want to create fibers by ourselves. For that we have fork or forkDaemon (that is detached from its parent fiber).
Other times we want to interrupt the fibers that we have created when they no longer serve their purpose (usually this fiber will execute some repeatable effect).

But it’s important to note that sometimes fibers are not interruptible! Examples include ZManaged acquire and release scope (The thinking is to make sure resources are acquired and released in a safe manner) and also when you specify uninterruptible:

criticalEffect.uninterruptible

In the example below the Managed.acquire scope is used to fork a new fiber that repeatedly sends out “heart-beats” to some other server to keep the communication alive.
The Managed.release scope is used to interrupt this (when the application ends)

The problem is that this fork cannot be interrupted, so the application will never stop and will continue to send out heart-beats!

The way to solve this is to explicitly tell ZIO interpreter that this fiber should be interruptible:

Managed.make(sendHeartBeat().fork.interruptible)(_.interrupt)

Shorthand syntax for this code is:

sendHeartBeat().toManaged_.fork

As toManaged_ creates an interruptible fiber, unlike Managed.make.

Thank you for reading!

Checkout the prequels to this post:

If you’d like to get updates on my experiences with ZIO, follow me on Twitter and Medium. If anything is unclear or you want to point out something, please comment down below.