One of the most memorable parts of “The wallet” app is a 3D-picture of cards & coupons with glares on the surface and light sources along with the option to turn the objects around. Employees on the interviews tend to ask about how we managed to integrate such a feature and, since this topic causes much interest, we decided to talk more about it. This option Is not something you normally find in a business-related apps, but our idea is not a complicated self-made 3D-engine. And if you have tasks similar to that and you find them quite unclear — we’ll show you how it works.
The Backstory
Originally 3D-rendering was made as a cross-platform solution powered by OpenGL. This solution is still being used in an Android version of the app.
It’s worth mentioning that 3D is not only the card itself, but also the list of the cards. In future we are planning to combine the list of the cards with the function of viewing the list of the apps from the catalogue so that we decided to abandon use of this solution. From now on for viewing the list of cards we use UICollectionView with a custom layout. Btw, our Android team made a report for a conference on this topic: “The wallet app: how we animate the cards” (RUS)
In 2018 on WWDC Apple has announced that the following iOS versions will no longer support OpenGL and recommended moving to Metal. A detailed comparison of these 2 technologies can be found in the Metal for OpenGL Developers report from WWDC. Even though OpenGL is already deprecated for several years, it’s still available for use in iOS 14; quite clear that it didn’t have any update ever since. So we estimated our abilities and time we have to spend on transferring the existing solution. We decided to use the fresh technology and try to write the MVP. The simplest would be use of some 3D-ready engine that supports both Metal and OpenGL. This would make it possible to set up the parameters via the visual editor and would also simplify working with the models, textures, camera lighting and animations. But pulling the engine into the project for a rectangle with a cool texture is not something we can afford. For example, use of Unity would increase the size of an app (according to the user manual, minimum app size is 12MB which is a 20% increase for us) and also cost us $200 annually with a logo on a splash-screen(can be removed for the more expensive plan) along with increased loading time. If we look at the open-source solutions, like LibGDX, urho3d, we discover that Metal is not supported there. Here a native solution by Apple comes to rescue — the SceneKit which has no such shortcomings as described above(it does not affect app size — empty project comparison showed just 1KB app size increase, it’s free, has a visual editor and many more).
We had quite nice results within a single day of testing and we decided to move on to the production with it.
SceneKit
SceneKit is a high-level framework that operates above Metal or OpenGL, simplifies work with the 3D and allows to easily add the animations, physics, particles, realisting rendering based on physics. What causes our interest is the last point.
SceneKit supports multiple concepts for visualizing materials — these are: blinn, constant, lambert, phong and physically based. The details about each of those can be found in this manual. However the real comparison of each approach is best shown on the following picture:
As you may have noticed, physically based here is the obvious leader
Physically based rendering (PBR) is the materials rendering concept based on real world physics. This is covered in detail in Advances in SceneKit Rendering from WWDC. Shortly, the idea of the approach is that any surface contains 3 components: diffuse, metalness and roughness. Let’s take a look at each of them on an example of a golden surface:
Diffuse shading describes the amount and color of light reflected equally in all directions from each point on the material’s surface. The diffuse color of a pixel is independent of the point of view, so it can be thought of as a material’s “base” color or texture.
So, as long as gold reflects uniform light across the whole surface, the diffuse texture will look like this:
This property generally describes aspects of a physical surface that together form metallic or nonmetallic appearance. Lower values (darker colors) cause the material to appear more like a dielectric surface. Higher values (brighter colors) cause the surface to appear more metallic. So because we draw a golden texture without any additions and gold is considered a metal, then the metalness texture should be flat white:
Describes how smooth the surface is.
Lower values (darker colors) cause the material to appear shiny, with well-defined specular highlights. Higher values (brighter colors) cause specular highlights to spread out and the diffuse color of the material to become more retroreflective.
Gold is almost perfectly smooth material — that’s why roughness texture is close to black and for adding texture defects we should add some light coloured parts.
By adding all three together we get this:
The connection between metalness and roughness can be seen here:
Another important fact: This technology is available in the SceneKit starting from iOS10, but we discovered that because of moving to Metal v2 the final image heavily differs and we now have to support both configs. At that point the minimum supported version was iOS9. Having 3.8% users on iOS9/10 around 55k people(which is about the size of a small city) it was a hard decision to abandon supporting their devices. After we weighed the pros and cons(the major was the layout as iOS11 began the era of iPhone X screens and safe area, as well as lower testing load) we decided to put the minimum supported iOS version to 11.
Implementation
The basic element of SceneKit is the scene (SCNScene) so we have to add it first. The manual says it’s better to save the scene and the texture images to an Asset Catalog — a folder with .sncassets extension. So we add it at File > New > File… > Resource > SceneKit Catalog. Further on all files connected to SceneKit we will place there. We have two simple ways to add a scene: You could either add manually by programming it and add objects from the code or you can use the Scene Editor and add scene there by File > New > File… > Resource > SceneKit Scene File. We chose second option and here we have a scene with a camera
Besides these two options you may also upload scenes from a file or by url.
Next step is going to be adding a 3D model to a scene. Ready-made primitives can be used for that:
Not likely that you are going to need it. Of course, one can make an object of primitives, but SceneKit does not support object texturing and one can’t edit texture coordinate — viewing is only available. That is why it supports third-party models from external editors. We used Blender. The basic format for uploading a model into SceneKit is DAE (Digital Asset Exchange or Collada). At the upload stage you can convert it into SCN format or you can do that later through Editor > Convert to SceneKit file format (.scn). Reverse conversion is also available.
The next step is adding a light source — we can do that manually. There are several types of sources available:
Another interesting option: add an occlusion texture and generate light sources automatically based on it. In this case it can be any shape. Our sample looks like that:
We add a texture into the project, then use Background and Lighting as an Environment in the scene preferences. It also can be used as a Background image. It is important to note that the textures shall match the requirements and they can also be in HDR. For a spheric texture the width to height ratio must be 1:2, for a cube it must be 6:1. You can find more about formats in the manual, p. Using Cube Map Textures.
As a result we get this glare from a light source and the scene now looks like this:
Now we gotta add the material. As mentioned above, for PBR we need 3 texture types: diffuse, metalness and roughness.
So because we wanted to achieve a metallic reflection effect on a part of the card — like an aluminium foil, we color these light when making a metalness texture to make it look more like a metal. Main background remains full black since the card surface is mostly plastic. For roughness we do the opposite: color the objects black to achieve mirroring reflection while leaving the background white to add a matting effect to the surface.
Moving on to the material preferences: in Shading, select Physically Based and add the textures to the project. One of the key parameters for textures is intensity — a number between 0.1 to 1.0 and, depending on a texture type, it gives the texture different features. E.g. for the normal property reducing intensity makes the surface appear more smooth, while for metalness and roughness textures — general dimming. This way we can control the effect of the texture over the final result.
Plastic cards are usually covered with a protection layer and at a closer look there can be found some “iridescent dots”. To achieve such an effect we need to use multilayer material, but SceneKit does not support that. We found a tricky solution for achieving a similar effect: we use a normal property texture with a randomly generated white noise:
On the scene we can see the final result of our efforts. It is worth noting that what we see in the Scene Editor will perfectly match the result on the phone screen:
Now we have to put the scene to the screen. For that we are going to use an object called SCNView — the successor of the UIView. It can be called from it’s code by mentioning our Scene or just by adding it to the View in the UIViewController via the storyboard. For debugging we put allowsCameraControl property to “true” to get the control of the scene camera and showStatistic to reflect current FPS and polygon number on the screen.
Finalize the project and see the end result:
Apple Watch
Another advantage of SceneKit is Apple Watch support. Just several steps are to be made to view the cards on the watch. Let’s add our scene and all it’s materials to the Target extension Apple Watch.
To view the scene instead of SCNView we use WKInterfaceSCNScene. Let’s add it to the storyboard in our controller and add a property:
The workflow with WKInterfaceSCNScene looks the same as SCNView, except for a few moments:
- Automatic camera control is not supported so in order to animate our scene we add an infinite card spin animation.
- Occlusion textures are also not supported in HDR, so this time we are also going to use the autoenablesDefaultLighting, which adds omnidirectional light source to the scene.
As a result we get this:
Problems we faced in the production release
Now that we needed three textures instead of one and it took us more time to draw each manually, we decided to make it another way. As a metalness texture we use a black&white image with a high-contrast from the diffuse texture and then we darken light parts. After that we additionally darken the whole texture using intensity property. Reminder: the lighter the surface we use, the more metal it looks and vice versa — the darker it is — the more matted it looks.
As a roughness texture we use adjusted numeral value with slightly lowered intensity, as long as most cards have a uniform surface. Reminder: the darker the color is — the glossier the surface appears.
As long as noise texture is used for normales, we generate it instead of storing it within the project:
Besides all we had to change the direction of the light sources depending on the phone position. It looks quite simple here: calculate the angle of the device using the CMMotionManager and adjust the light sources. But the occlusion texture does not support spinning so we had to put both camera and a map into a separate object and spin it all together. This creates an effect of lightray direction change.
In fact use of the occlusion textures caused most problems for us during the production. For iOS versions below 12 the scene simply didn’t render properly with enabled occlusion texture for some devices (from iPhone 6 to iPhone X) while we weren’t really able to track the exact reason for that. We also received a load time increase with iOS14 release so we added the light sources programmatically. Sadly, the final result looks not as interesting there.
In order to no to freeze the app, the objects can be loaded in the background using prepareObjects:withCompletionHandler: method, but there is one important thing here: despite calling completionHandler one can suddenly see an empty scene for a moment as long as no frames were pre-rendered. In order to avoid that, we have to wait for delegate in SCNView to render the scene — SCNSceneRenderDelegate in renderer(_:didRenderScene:atTime:) method.
Summary
As it usually happens, Apple provided the developers with a high-end tool for working with the 3D and, as a result, by making a small effort we managed to create an MVP that allows us to build a production solution.
Pros:
- Development speed
- Minimum app size change
- Independent from the third-party solutions
- Simple code support, no need for deep integration for making minor changes
- Simple use of camera & lighting
- Supported animation
- Integrated physical engine with supported particles
- Full Apple’s ecosystem support, including Apple Watch
Cons:
- No multiplatform support
- Hard optimization. Use of Metal would give more opportunities for this
We can see that pros obviously dominate over cons for this approach. That is why, when you need to make something relevant and have no time to deeply inspect low-level optimizations(and you don’t need it to be multiplatform), you will definitely find SceneKit the best solution so far. Plus, it is easily compatible with the UIKit and has a quite high development speed. We found the implementation quite an easy process, unlike it may seem. However, it does make a wow effect for an average user and is being the most impressive feature of the whole app.
Btw, we regularly post more about our tasks and solutions in our Telegram channel (RUS)
