Remote hosting of 3D models
We are working on an augmented reality application for a furniture manufacturer. You can virtually place furniture in the space you want to furnish, so you can see exactly what it would look like in reality.
The basic requirements are:
- Detect a flat surface, usually a floor, to be able to place 3D models;
- Render models as realistically as possible;
- Change size of models (if different sizes exist);
- Change texture on the placed model.
Aside from this, we also need to support a huge number of models and textures. They need to be added or changed anytime, without publishing a new version of the app, or making the app large. That was the challenge we faced — and these are our experiences.
As we started experimenting with ARKit, we kept the models in .scnassets. We researched what the models should contain, how they need to be aligned, which lighting models to apply, etc.. This was a good choice for testing changes on models. Then we realized that this was not a solution for our app. Drawbacks are the size of the application with all the models and textures. Also, the inability to add, change or remove a model without releasing a new version of the app. The next step was to found a solution for keeping assets somewhere else and fetch them when needed. After some research, we found Apple On-Demand Resources. It seemed like a good solution for our case.
Apple On-Demand Resources
On-demand resources is out-of-the-box solution from Apple. It separates assets from the application and fetches them when needed. Assets are uploaded to the App Store, along with the application, but not downloaded to the device when the app is installed. Instead, the application downloads them when necessary. That solves our first problem: the size of the application on thee App Store.
So we implemented this solution and followed the Apple documentation. We enabled On-demand resources in our project, by changing Build Settings for “Enable on-demand resources” to “Yes.” Our next step was to assign tags to our assets. A tag is key for a group of assets on a remote server which is downloaded together. That means that when we request assets with a certain tag from the app, we will download all assets with that tag and not one by one. It is possible to assign one tag to one resource, but Apple does not recommend it. There are two ways to add tags:
- Adding assets in Resource Tag
- Writing the tag in the tag property of asset.
Assets with tags are deployed to the App Store. If the application is not distributed through the App Store, you can store them on your own web server. When we request assets in the app, it checks if they are already on the device. If not, it downloads them. That prevents duplicates, saves memory space and network usage. This is especially useful when the user is on a mobile network.
Our problem was that there was no certainty that the assets will be permanently saved on your device. They could disappear anytime after the app is closed, and we needed them to be saved through multiple runs of the app. It is possible to prioritize preservation and tell the system which resources have a higher priority in preservation. But this is still is not a solution. Also, the only way to upload new resources is to publish a new version, along with new assets. Than we need to figure out a new solution and forget the Apple On-demand resources.
Custom Solution
In the end we agreed that we need our own, custom solution to solve all existing issues. We did not want to build the whole backend for it; instead, we used Firebase Realtime database and storage. It could be any storage or any JSON-like database, but we decide to go with Firebase. In the final solution, we changed our storage to Amazon, but kept the Firebase database.
Next, we had to define the structure for the database. It had to have all the necessary information, as well as the ability to add, change or remove models or textures. We had to take into consideration that products come in many sizes, with many combinations of materials.
Our structure looks like:
We have a array of products, which contains data common in every variant. The product contains a array of variants. They are different sizes of models with model url, dimensions and url for shadow. Also, the variant has a list of nodes with node name and textures, which can be applied for that node. Textures have urls for diffuse, metalness, roughness and normal and for the display name. All assets are uploaded to storage. The urls for download are entered in firebase realtime database. One separate node in database is materials. It has all the possible materials grouped by categories with thumbnail. We keep it in a separate node, because we do not want to download the same thing multiple times for every variant and duplicate the data in the database. We have the version node as the parent node for materials and products. This way. if the structure changes in the next versions of the app, it the previous versions will not crash.
We started with Firebase SDK to retrieve the data from database in the app. But that was not an option for us, because our data is nested in many levels. The data snapshot can not be mapped to objects. To avoid nested loops, we created Cloud functions which return JSON data from the database. Cloud functions retrieve version of the app as a parameter and return database node for the given version. Data is directly mapped to objects with JSONDecoder in Swift 4. We defined classes which correspond to data structure in the database and inherit Decodable. Also, if class properties are not the same as the keys in the database, we can define Coding Keys to match it.
Implementing Cloud functions give us the ability to work with data before it is sent to application.
Firebase Realtime database is a schema-free NoSQL database. This way, there is a risk of missing fields or typos in the keys and that can cause the app to crash. So we marked all the values as optional and handle cases when something is wrong or missing.
For caching, we keep files with the name of the url hash. On download, we check if the file already exists so we don’t have duplicate downloads. This is done for everything in assets. This way, new files download only when the hash changes in the database.
Conclusion
If you are building an app with a lot of assets, such as images, videos or 3D models, you should keep your content remotely to reduce the size of the app. Depending on how often the assets change, you can use your own web server or Apple-On demand resources. What you choose depends on your needs. For us, Firebase had everything we needed. There are a lot of options out there and all of them have their pros and cons. You might need a custom backend instead of Firebase, or something similar. Do your research before implementing. Happy coding.
Originally published at www.rubicon-world.com.
