Code-generation with Sketch
How we accelerated our weekly internal deployment process with the help of Sketch.
TL;DR
“With the help of generated code from our Sketch code brewery we automated slow-moving internal deployment creeps. This is not some sophisticated article about the Sketch API but more or less a summary on our experience of pushing a yet digital but not fully automated process to the next level.”
Introduction
The prequel of this article is a small introduction about who we are and what we are up to and can be read here.
Rationale
For our smartphone applications we need an overlay of a test cassette so that the user can align the smartphone properly to a predefined position. Each of our supported smartphones needs to bring a torch onboard. These differ in light intensities, colour (we’ve seen warm to cold whites and even greenish colours) and emission characteristics (diffuse point lights to more directional lights).
Additionally, we receive new test cassettes on a regular basis for which we need to create overlays for, to have our laboratory do stability studies and phone model releases. For each new test cassette an overlay needs to be created and quickly deployed for an internal validation process and further hardware development. For this process we developed an internal development tool called Cappuccino ☕. It serves as a data hub for our laboratory to quickly gather lots of data from smartphone test cassette captures. It provides a web frontend connected to a backend, as well as simple to use capture-only smartphone apps for Android and iOS. We established weekly deployment cycles for these apps.
Problems
History
In the ancient history of Immundiagnostik, overlays were drawn by hand (i.e. actually scratching them into rocks using a hand axe in a dark cave with only dimmed fireplace lighting) — or being a bit less dramatic: were manually created with a time-consuming procedure in Photoshop. Dozens of those overlays needed to be created as each test cassette is different — it was a huge sluggish process.
Overlays have been served as pixel-based images using the png file format. New ones have been created by using an old overlay and destructively editing them. Thus, we couldn’t re-create existing images. Minimal adaptions to existing overlays resulted in hours of pushing pixels at the correct positions. Even worse: the overlays have been saved as flattened raster graphics instead of full-layered Photoshop files. As a result, image quality loss while zooming the overlay images in an app and annoying editing workflows in Photoshop (cutting and moving pixel regions destructively over and over) were predetermined. As stated in the introduction: we were in need to create a new overlay for each combination of smartphone device and test cassette — and this task became one of the most hated ones!
Sometimes one had to quickly create an overlay for testing of new cassettes in the laboratory — even if they would never actually end up in a final product draft. We did not have a planned process for this, which quickly led to an annoying and intrusive task, as overlays sometimes were needed as quick as possible.
Annoying and intrusive process
But let’s make two steps back: why is that such a huge time consuming and error-prone process?
It is not only the overlay which has to be drawn and positioned on the smartphone’s screen, but additional device-specific configuration is necessary, i.e. the app needs to know where the barcode is positioned.
The test cassette consists of different parts, one that is also important is the viewport. The viewport shows bands with different intensities. Those bands are the result of a biochemical process and are used for result calculation.
Hence, we need to know where the barcode and the viewport is located. The barcode is used as anchor point to locate the viewport relatively. Additionally, the barcode is used for the batch data retrieval, like the test cassette type and some validation information (e.g. expired batches).
Setting up the scanning rectangle positions was done manually, meaning you had to find the pixel coordinates inside Photoshop and copy-paste them into a web frontend.
Additionally, as we used pixel-based images, animating parts of the image wasn’t possible at all and — as mentioned beforehand — hardcoded sections like scanning and viewport rectangles were necessary.
Solution
We elaborated on Sketch’s exceptional plugin API and developed a custom plugin with Javascript, that produces app-specific code, generated utilising string templates. As of now we use Sketch for drawing overlays on a vector-shape basis and export them using our plugin.
Diving into Sketch Plugin API
A good start for Sketch plugin development is probably the introduction from Sketch. Additionally, there is a very solid API documentation nowadays. Back when we developed the plugin, there were only little information published, so we had to rely on the outstanding sketch-dev-tools to find our path through Sketch.
At first, we needed to find out how to access an Artboard and the containing elements. Then, we had to clearly define Shapes to be able to process each part of such an overlay in a proper way. So, these are the ingredients we need: an outline for the masking (and of course the outer boundaries of the test cassette), the scanning rectangle for the barcode, and the position of the viewport for the algorithm.
Actually, there are a bit more elements involved related to our algorithm, but for simplicity we just leave them out.
It turned out, that Sketch provides a nice interactive way of developing plugins. You have a full developer console that enables interactive discovery of the document’s structure, which leads to an iterative workflow with direct feedback.
The final plugin
The end result of our plugin is twofold, we have the generation of source code in either Kotlin or Swift which produces app-specific code for drawing an overlay using CoreGraphics on iOS and Android’s Canvas Graphics. Even better, we are able to generate test-cassette specific batch data, i.e. sensitive properties for our algorithm to get properly executed and finally quantify the result of the sampled test cassette.
The plugin itself is very tiny: we defined a clear structure of an overlay’s Artboard and wrote some methods to extract important properties. Then we created two overlay templates (Kotlin, Swift) for the respective operating systems. Basically, the template overlay class is very tiny and inherits from an existing GenericOverlay class. Hence, the generated overlay uses no real logic.
The final generic-overlay-swift.js template is a bit longer, as we need some additional logic to only draw the parts, necessary for the user, and to hide the parts which are related to internal business logic, like the scanning rectangle for barcode recognition or rectangles used for debugging purpose.
Reading out Shapes is relatively easy. It looks a bit nested until you reach the desired properties, but basically it is just iterating over each children until you find some Shape (e.g. RectangleShape). In our case, the Shapes use relative positioning to their parent, so you need to take the overlying ShapeGroup into account. Here we exactly knew how our Artboard will be structured, so it looks a bit less ugly.
Exporting an Artboard is primarily gathering information from the Shapes, filling the template and writing the data to a file. The interesting part is that skpm exposes the complete Cocoa framework. Thus, you can use whatever classes and methods you are familiar from OS X development with, but be aware: as Sketch uses the Objective-C runtime, you need to stick with the naming of that runtime (and not Swift 🙃).
To ease the path for the creation of new overlays, we also added a template for new overlays that contains the most important parts for our code generation process. This was realised using the react-sketchapp which helps with the generation of assets in Sketch. You basically implement design in code as React components and render them into Sketch. “react-sketchapp evolved out of our need to generate high-quality, consistent Sketch assets for our design system at Airbnb.”, for a solid follow up click here.
The current process of overlay creation
In the end, we delivered a solution for coworkers to quickly add new overlays to the party 🎉 without in-depth-knowledge of each application.
Conclusion
With our generative approach, a new test cassette as well as parameters for our algorithm can be created and exported within seconds.
Most important feature
It feels awesome having created this plugin. It improved our workflow and even enabled versioning of our overlays using git (as well as the whole overlay creation process, the Sketch plugin).
Sketchy
It was a bit of a Sketchy 🤔 experience to implement the whole plugin as during the development process a new Sketch version arrived. After that upgrade we couldn’t use console.log() any more (👉 cf. issue). That moment was a small downer.
After a while we figured out that we could use NSLog() but still we did not see any logging output in the terminal. However as NSLog() should directly log to the OS X Console, we happily found the desired output there.
Additionally, we have to state: the sketch-dev-tools are amazing. We were missing a printed API documentation as well as some more sophisticated code examples, but it was still a rewarding experience. And for fairness reasons we should add, that the API documentation is much better nowadays 👏🙏.
Another hint on Sketch plugin development is the collection of recipes from turbobabr.
Future outlook
There a tons of ideas on how to improve the above described workflow: e.g. we may want to try machine learning or image recognition to automatically create an overlay’s outline shapes from photographs. Additionally, this could also help to to verify the user’s input for automatic overlay testing, i.e. look for specific markers in the viewport or trying to identify the barcode within the scanning rect, so that only feasible overlays are committed and deployed.
Sketch could also be used to generate illustrative graphics for the user guidance we provide in each app, i.e. generate the code for rendering an aesthetic/appealing figure of a test cassette that could be used for guiding the user on how to overlay the smartphone camera over the real subject.
We would be happy to see what you may bring to the table and of course are very open to feedback and questions. Don’t hesitate to get in touch with us.
