Building 3D Avatars in iOS using SceneKit
While playing around with your smartphone, there must have been a situation when you saw a 3D object on the screen and you wondered how it was constructed under the hood.
In this article I will explain the key concepts of modeling a 3D object in the SceneKit and mention the requirements that were requested by our client — The Football Company GmbH. Also, I will also describe challenges that were resolved on the way to a complete solution.
Here are some key functionalities requested by our client at the beginning of our 3D journey:
- Make a 3D avatar that will be shown in multiple places in the application
- User needs to be able to dress the avatar and exchange all kinds of different clothes on it
- Avatar should support both genders as well as multiple skin colors
- Avatar needs to support custom gestures like taps, rotation, zoom, etc.
- User should be able to place his avatar inside the real world
At first sight this sounds like a complicated thing to do, and it’s not easy, trust me, but we have managed to do it, and here’s one screenshot of our masterpiece!
Choosing the right file format
The first thing that was discussed among our teammates was choosing the proper file format for 3D objects. After some research we found out that Scene Kit supports these file extensions by default: DAE, ABC, SCN, OBJ etc. Keep in mind that the client requested an application for both mobile platforms(Android and iOS), so we needed to choose a file format that will be supported and used on all existing mobile devices. After some googling and consultations with the 3D design team, we agreed that the best thing to do is to try a couple of different file formats and see which one matches our requirements. We choose the next file formats: OBJ, GLTF and GLB. Designers exported all kinds of test models to the mobile teams and we agreed that we were going to use GLTF files. There were couple of reasons for choosing this file format and I will list some of them:
- GLTF supports links to the external files like textures, shaders and animation data (unlike GLB format)
- GLTF is way more efficient in terms of size then OBJ file
- GLTF supports scene graph hierarchy
The image above represents the structure of one 3D object constructed using a GLTF file. The biggest advantage of this file type is that textures are separated from the node structure and therefore you can link all different kinds of images to your base object. The real power of this can be seen by taking a look at the bottom part of the first image where all different kinds of sweatshirts are constructed using the same base GLTF file and different external textures. Maybe you are asking yourself:
Why doesn’t every sweatshirt have it’s own GLTF file?
When you are constructing something like a 3D object, you need to take care of memory management and loading time. Therefore, with having one GLTF file for all of the items from the same category, you are saving tons of memory. We also implemented LRU (Least Recently Used) cache for all of the GLTF files and their textures. With cache implementation items are stored in the device memory and, by doing so, they are loaded faster.
Scene, lights, camera 📸
In this section I will explain the basic scene setup that was constructed for the avatar on the first image. Firstly, there are 4 different types of lights inside the scene kit:
- Omni light → point light which illuminates all directions from a single point
- Ambient light → light that illuminates from all directions with equal effect; orientation and position of the light node can be omitted
- Directional light → this type of light illuminates from one direction, so the position of its node is ignored
- Spot light → this type of light illuminates a cone-shaped area, the orientation and the position of the light are defined by its node’s properties
For the scene we used directional light in combination with a custom HDR file that was used as a background for the whole scene. On the first image you can see how beautiful shadows are constructed due to the directional light position in front of the avatar. Also, eyes of the avatar are made from glossy material and because of that, light can be seen inside avatar’s eyes which gives a more realistic look. You wouldn’t be able to see your avatar inside the scene if the camera wasn’t there, so make sure to add the camera and position it wherever you want. You can also play with some of many camera settings and get some really fascinating results. We implemented custom gestures so the user is able to zoom and rotate the avatar. Avatar rotates by default and can be stopped with a single tap on the screen. The avatar rotation and items exchange are displayed below:
Morphing
Morphing was by far the most complicated topic in the whole development process.
3D objects can have multiple geometries (imagine those like states) and a set of weights associated with each. When you apply weight to some geometry, your 3D object changes it’s appearance and can look completely different than the initial one. An image that can describe this in a real world scenario is one from the Apple’s documentation:
All objects in the previous image have exactly the same GLTF file, but inside the file there are 3 different geometries that you can change across. Transitions between those geometries can be animated as well. If you still can’t see where this morphing thing can be useful, I will show you in a moment. Imagine having your avatar fully dressed and your client presents a new collection of different scarves for the avatar. Everything sounds good to you, but when you put your new scarf on the avatar, as well as some hoodie, you can see some clippings:
This is where morphing comes into play. If you have multiple geometries both for the hoodie and the scarf, you are able to switch between different states of each item and fine-tune it to remove the clippings. We have developed our own algorithm for automated adjusting of geometry weights to ensure there is no clipping between items. The result can be seen on the image below:
I encourage you to invest some more time in morphing and you will see how cool things can be achieved using this feature.
“Don’t settle for average. Bring your best to the moment. Then, whether it fails or succeeds, at least you know you gave all you had.” — Angela Bassett
Avatar in the real world 😍
One of the things that our client requested was a showing of a 3D avatar in the real world. So we did it. In the next video you can see me walking down the park when suddenly my avatar showed up.
Showing of the avatar in the real world was done by using the AR Kit. AR Kit is Apple’s framework used for showing all kinds of different 3D objects inside the real world using the phone’s camera. If you are playing with 3D objects, this feature is a MUST HAVE inside your application.
Conclusion
In this article I explained the complete development process and some difficulties I encountered while creating a 3D avatar from scratch. There are a lot of things to do in the middle of each step I mentioned in this blog post, but I left those steps to the ones that want to learn and try some things by themselves. I hope that you enjoyed reading this blog and that you learned something. If you have any questions feel free to reach out to me using the comments section. At the end you can download the application and play with your own avatar. Here are the links to the App Store and Google Play:
Thank you and have fun.😇
Krešimir is a valuable member of our iOS team.
At Azikus, we design and develop top notch mobile and web apps.
We are a bunch of experienced developers and designers who like to share knowledge, always staying up to date with the latest and newest technologies.
To find out more about what we do, feel free to check out our website.
