Advanced FP for the Enterprise Bee: Optics

Sunlight shining through honey

Introduction

This is the sixth article in our exploration of Advanced FP with Kotlin and Arrow. This time we will be exploring the fascinating topic of Optics.

Optics requires a more complex example. So, instead of a made up case study, I’ve taken a real one from my day job as a trainer / coach at Instil. It will take longer to explain than a toy problem, but hopefully better demonstrate the power of Lenses (and other Optics). As always all the sample code is available in this repo.

The Case Study

At Instil our training team makes heavy use of JetBrains Space. We create a dedicated instance for each virtual delivery, which gives us the ability to host repositories, mentor via chat channels, post blogs etc… We were featured as a case study by JetBrains at the product launch.

Creating one instance per delivery ensures confidentiality, but it does require a lot of manual setup. To automate this we have created a Kotlin DSL for describing Space instances.

Here’s a simplified example:

This was designed using the standard Kotlin pattern for creating DSL’s, as documented on the language website. If you would like more details, this talk we presented at Kotliners 2020 describes the process in detail.

Here are two classes from the DSL. Note that the functions to implement the DSL (repo and project) and the fields to hold the data (projects and repos) are contained within the same types:

Normally we would interface with a RESTful client (written in Spring Boot) that uses the Space HTTP API to populate the instance with resources. Here I’ve added simple toString methods to prove the data has been captured:

Kotlin Programming for MegaCorp
Current profiles:
 Jane Jones at Jane.Jones@megacorp.com
 Pete Smith at Pete.Smith@megacorp.com
Current projects:
 Project 'Course Examples' (PROJ101) with repos:
  http://somewhere.com
 Project 'Course Exercises' (PROJ202) with repos:
  http://elsewhere.com
Current blogs:
 Welcome to the Course
  welcome.md
  Some additional client-specific content
 Setting Up
  setup.md
  More client-specific content

The Sample Problem

Let’s say that we need to support alternative formats for describing the content of the Space instance. Perhaps we need to create a Web or Mobile version of our application.

The first stage would be to create a set of entity classes that are independent from the types that make up the DSL. As aspiring functional programmers we want these entity classes to be immutable. Any attempt to modify an entity should cause the creation of a new one. The state of an existing entity should never change.

Creating the Entity Classes

Here’s how we might separate things out. For clarity the names of our DSL types are now prefixed with Dsl:

Our new entity types are declared as data classes, so the compiler will add extra functionality on our behalf. As discussed all the fields will be immutable:

Populating the Data Structure Manually

Our task now is to write code that creates a graph of entity objects, based on an instance of the DSL.

Let’s start by defining some simple extension methods:

Note that these methods are only making a shallow copy. For example when we create a Project from a DslProject we copy the name and key, but not the nested repositories.

The tricky part of the job will be implementing a deep copy of all the nested information. Here’s a first attempt:

This is overly long, but not tortuously so. We are making good use of the copy methods generated by the Kotlin compiler for data classes. This saves us an awful lot of work.

The problem is that this example is just a subset of the data we wish to capture in the DSL. As we expand our implementation we will have more and more nested calls to copy methods. Our createInstance function will grow, and be increasingly hard to maintain.

The Need for Optics

This is a general problem in pure functional programming. We want our code to be free of side effects and (hopefully) easy to reason about. Ideally we want our functions to be referentially transparent - whereby any function call can be replaced with the corresponding result. This drives us toward modelling our problem domain with immutable types.

In one sense this is not controversial, after all Strings are immutable in most modern languages, and immutable collections are the default in Kotlin and Scala. But, once we start creating deeply nested data structures, the need to make a deep copy on every modification will become a burden. This is the problem addressed by Optics.

Introducing Optics

Let’s leave our createInstance function alone for now, and start with three simpler examples:

Here’s the Hello World of Optics. When we say Instance.title below we are creating a Lens object. This will enable us to make a deep copy of an existing instance, with a single modification.

The metaphor is that we are focusing on the title of the Instance as we make the copy — everything else will stay the same:

When we call this function, and print the result, the output is the same as before. Except that we have successfully changed the title:

Space Instance - 'Kotlin Programming for MegaCorp - as taught by Instil'

It’s important to note that:

• Lenses can be created at an arbitrary depth. So in a purchasing system we might say ‘Order.customer.payment.rating.discount.percentage’.
• The entire structure is copied for us, regardless of the complexity.

This example applies to any chain of single objects. But what if we were working with collections of values? Fortunately Arrow provides Optics for working with lists, sets and maps.

Let’s copy our Instance, but alter the URL of the first repository within the second project:

When we print out our new Instance, we can see the change has been made:

Projects:
   Project 'Course Examples' (PROJ101) with repos:
     http://somewhere.com
   Project 'Course Exercises' (PROJ202) with repos:
     http://nowhere.com

The final example addresses a real world problem. In the office, when we receive lists of delegates names, the capitalisation is often inconsistent.

Space doesn’t care if the username for a profile is ‘Jane.Smith’, ‘jane.smith’ or even ‘JANE.SMITH’. But when coaching large groups this can cause any number of trivial, but time consuming, mistakes.

So we create an Each, which will allow us to iterate over the names within profiles. For this demo we will uppercase the forenames:

Once again the result is what we would expect:

Profiles:
   JANE Jones at Jane.Jones@megacorp.com
   PETE Smith at Pete.Smith@megacorp.com

Configuring Optics

In order for this wizardry to work we must do four things:

• Include the Arrow Optics library in your build file
• Decorate the entity types with the optics annotation
• Add a companion object, if one is not already present
• Use Arrow’s collection wrappers (SetK, ListK and MapK)

Here’s an example type:

When you run your build file Arrow will use the Kapt annotation processor to add the different Optics (Lenses, Prisms, Traversals etc…) to your types. Note this means your IDE may not be able to see the Optics functionality till you (re)run the build file.

Returning to the Problem

Let’s revisit our initial problem of performing a deep copy, from the existing DSL types to the new immutable entity ones. The first thing we can do is replace calls to copy with Lens objects:

One possible variation would be to use the Cons lens, which inserts an item at the beginning of a collection. For any of our entity types we can say ‘obj.cons(things)’ — where obj is the entity object and things is a collection:

Because we are only inserting at the top level of our structure this doesn’t make a lot of difference. But the more layers we add the more useful it will become.

However we can already spot a useful abstraction. We can write a generic insert method that copies a structure of any complexity, whilst adding an extra item to a list:

Once we introduce this new abstraction it becomes apparent that every usage of forEach should actually be a call to fold. This reduces the number of assignments and hence the overall complexity:

This new version is much shorter than the original (from 37 to 25 lines). But more importantly it is simpler in structure and will scale better as we expand our DSL.

Conclusions

The further we go into functional programming, the more we will want to make our data types immutable. The more deeply these types are nested the greater the complexity involved in modifying them. Kotlin provides some support through data classes and copy methods, but a lot of boilerplate code still needs to be introduced.

An Optics library generates this boilerplate code for us. As a result we can easily make copies of deeply nested objects, with one or more changes applied. In Kotlin, the Arrow Optics library layers this support on top of the abstractions already provided by Arrow Core.

Thanks

Continuing gratitude to Richard Gibson and the Instil training team for reviews, comments and encouragement on this series of articles. On this occasion extra thanks to Maarten Balliauw, for coaching me last year on the Space design and API. All errors are of course my own.