From large GPS datasets to 3D animations — 4 key learnings for a procedural workflow with Houdini
As a technical artist and developer at Sapera, I’m constantly improving our workflow when visualizing data. When our Art Director Cartography came to me with a possibility to do that in his field, a team was put together to meet his challenge:
“As a non-coder, I want a tool that enables me to quickly explore GPS data, visually. I need to get to potential stories hidden inside the location and movement of the data (e.g. tracking data of animals). I need representation in 2D, but ideally 3D, and the possibility to show animations of movement over time. On top of that, I need a process and tools to then import such data to design the stories”
Houdini turned out to be the tool of choice for our project, optimizing workflow as well as visualization. In our case, we tracked data of animals, but what we learnt while bringing our project to life, can be transferred to any kind of GPS data visualization with Houdini.
That’s why I’d like to share 4 key learnings with you that we discovered in our workflow.
1. Understand the Data
First things first: Understanding data is the key point of every data visualization. We took migration datasets of penguins, birds, whales and others. Most of the time column specifications are already understandable. But we still have to transform the dataset into a more useable chart so that we don’t have to care about data errors in the visualization process.
Points you have to consider when working with GPS datasets:
- Date and time have to be converted to a more (developer) useable Unix timestamp format.
- One GPS file often contains more than one tracked object. So single GPS events may have to be fused into a combined path based on their local identifier (representing an ID of one tracked animal).
- GPS tracking isn’t always constant. Many days of tracking events may be missing and start/end times can be inconsistent. This has impact on the way of calculating and setting key frames in the development.
- Keep in mind: The sea level isn’t zero on every position of the earth, and the earth isn’t a perfect sphere (like our 3D model). So don’t be confused if the paths are cut by your earth asset or don’t fit perfectly. All height calculations of a GPS sensor are just approximations. Manual adjustments of your assets are mandatory.
2. Impact of File Size & Data Wrangling
When we talk about datasets and challenges, the file size is always a concern. We were working with datasets of GPS events larger than 1 GB.
We had to do some time remapping calculations on the data itself, so the first thing we did was use this opportunity to add and reduce information and minimize the file size. With a Node process, the dataset itself was reduced by removing all the attributes we didn’t need in the visualization process. But we realized that we need to add further information for path IDs to recognize connected data points and transformed timestamps. The tool also found and — if needed — excluded, broken or incomplete data.
We also included the option to export all paths separately to give the user the opportunity to import just the paths of selected animals. To reduce the points by interpolation itself would result in data distortion and that was not an option for us. All of this was done in a Node.js tool we created for this reason — it’s also accessible at https://ikarus.herokuapp.com/.
In the end we received a csv file of paths for the visualization process. Even with reducing shapes, the files, you still have around 60% of original file size (data dependent). Not many software applications can handle files of this size. Software like Google Earth or web-based methods like JavaScript or libraries like Cesium are not an option. Even popular programs such as Cinema4D have problems with animated large paths.
Our experience with Sidefx Houdini made it an easy software choice to explore our datasets — mainly because of its powerful performance and its handy workflow (including caching).
3. Final Setup and Workflow
CSV (Raw Dataset) → Node.js Converter (Converted CSV)→ SideFX Houdini
The main setup was pretty simple — the green geo nodes includes our static assets for the Earth, with high-res textures from NASA. Additionally, a control node gives the designer the option to switch (animated) between a sphere and a plane representation. The time node is a simple tool to convert the current frame to the actual date of the dataset and pastes the value to the cameras vcomment attribute. This way the designer has a better understanding of the real timing of the dataset while animating.
The Creep node cuts the path and can animate the movement. However, the obvious solution using CHOPs (Channel Operators) was our first try but because of the uneven distribution of time events in our datasets, we got wrong results. So we had to split up our paths and set the key frames by a python script. We added a parameter in our Node.js setup to return the percentage position of every single time event in relation to our full timespan. To make it even more convenient, we added the relative/local percentage in time of this point in relation to the timespan of its route. With this key information, it was quite easy to write a script, which sets up all key frames of a path on the right position.
To give the user more control over the design, we added some additional tools for adapting the visual representation. This setup was converted into an HDA and gets created for every single animal path by our main python control script. Furthermore, it links all the visual editor parameters of the HDA to a combined UI, representing the main control element.
4. Curated Proceduralism
Proceduralism in combination with Houdini’s non-destructive workflow is essential for us. It offers the designers — our users — an easy and handy way to manipulate and modify the visualization after creation. The more visualize options we integrated, the more extensive and complex the tool became. Bonus options like animated switch between 2D and 3D representation, lines, wall or wireframe representation, arrow animations, and a preset of render options were included and many more will follow. Besides that, we also implemented a heatmap visualization as an analyzing tool.
Conclusion
Houdini saves the day. It can handle large datasets, giving us a new approach to data visualization. It can be a huge improvement or addition in a more scientific field. Still data wrangling (partly automated) is an essential part so it was necessary to build a solid setup and to be able to handle dataset errors. Our biggest effort was to translate the data points to the correct representations in time and space.
In the end, the tool fulfilled our demands: Houdini offers more tools for exploration, visualization and even analyzation.
