The Ayotzinapa Case — A Technical Overview.
Forensic Architecture ( FA ), a research agency based at Goldsmiths, University of London, was commissioned by and collaborated with Equipo Argentino de Antropologia Forense (EAAF) and Centro de Derechos Humanos Miguel Agustín Pro Juárez (Centro Prodh) in order to develop an interactive web platform that aims to gather and present a comprehensive and cartographic collection of the events that took place in the night of 26-27 September 2014 around the area of Iguala in Mexico and led to the disappearance of 43 students from the Rural Normal School of Ayotzinapa, the murder of 6 people ( including 3 students ) and the injury of 40 more persons in a case with various degrees of evidence of direct and indirect participation and cooperation between criminal organizations, local police and other sectors of statewide security such as the military and federal police.
The aim of the platform has been twofold, on one hand attempting to indicate and highlight the gaps and inconsistencies in the official investigation carried by the Mexican state in regards to this specific case but also to demonstrate the possibility for alternative ways of action, for civil society, based on collective research, independent investigations and the use of modern digital tools and technologies.
ANSO, a new design-technology studio we have recently initiated, was approached by FA in order to assist with certain aspects of the development of the web platform which is essentially an interactive visualization tool for tracing, locating and cross-referencing the events that took place that night.
The main part of the application consists of an interactive map, a timeline and a filtering system where the user can trace the various types of incidents that occurred that night.
A video produced by the FA team that explains how the system works can be found here.
The specific part of the application was already at works from the FA team when we were approached to develop certain 3D aspects of the application. Three locations ( Juan Alvarez Y Perif. Norte, Palacio De Justicia, Cruce De Santa Teresa ) were of particular importance for the investigation that was carried out and FA produced photogrammetry-based 3D scans from parts of the specific sites with the aim to provide an interactive real-time view of them in the platform. The view had to also integrate, together with the scans, various 3D models that the FA team produced and represented different snapshots in time of the specific scenes describing the topology of the scene at that particular moment.
The original point clouds of these scenes consisted of several millions of vertices making it prohibitive to follow a standard 3D workflow of simply meshing and rendering the geometry. We needed to find ways to first reduce the poly count without loosing all the fidelity of the scans and second to figure out how to load and render this geometry in an efficient and relatively fast way. We really wanted to avoid having high loading times when entering the 3D scenes since the application had to operate reasonably even in moderate connections but also because long loading pages act only as a distraction from the subject that matters. In addition the 3D scenes had to be integrated with the filtering system, allowing the user to activate different Points of Interest ( POI’s ) inside the scene and the interactive timeline on bottom side which allows the user to explore the scenes based on time.
The first problem we tried to tackle was pre-processing the 3D point clouds. The desire was to use meshed versions of the scans instead of points so we started looking at ways to mesh and simplify those scans. Naturally, the first choice on this was to try the capabilities of Meshlab.
Meshlab is an open source tool which in its core lies a vast collection of algorithms that deal with various aspects of geometry manipulation. Part of this collection are two algorithms that seemed to fit exactly our needs : Screened Poison Surface Reconstruction and Quadratic Edge Collapse Decimation. The first one can be used for meshing point clouds and the second one can be used for simplifying meshed geometry ( aka reducing mesh poly count ). We started experimenting with this process and we soon realized that the final outcome would greatly depend on manual cleaning and tweaking of the original PCL’s. We needed to remove excessive vertex noise on the scans and fix normal direction in parts in order to achieve reasonable results during the Poisson surface reconstruction step. In addition Meshlab offers various parameters that can be adjusted when running these algorithms and can affect the final result. We experimented with these a lot. This was a time consuming but also rewarding process in order to understand what the limitations are when dealing with sparse point clouds that need to be meshed.
While continuing our search for alternative ways of compression and loading we finally came across progressive transmission and POP buffers :
Progressively Ordered Primitive ( POP ) Buffers is an initiative from Fraunhofer Institute and other research teams offering an alternative way from binary formats like glTF and the X3DOM binary container that supports progressive mesh transmission together with geometry quantization. You can find more info about the background of the project and the details of POP buffers implementation here. This seemed to us as an ideal candidate for our use case since on one hand it offers compression of the geometry, reducing the overall size that needs to be transmitted and on the other hand it allows for the progressive transmission of the geometry which means that the user gets immediate feedback and can interact with the scene while higher levels of detail of the scene are being loaded. We immediately went on a search for open source implementations of the algorithm in the hopes that we don’t have to implement the specification from scratch. Although the X3DOM framework is open source and implements the full specification of POP Buffers we found that the implementation there was greatly dependent on other parts of the engine which made it prohibitive in the timeframe we had to pull apart and reconstruct the necessary parts for our application.
Quickly we came across an open source implementation of parts of the POP Buffer specification that although incomplete seemed to partially cover our needs for this project. The specific demo included a POP buffer binary encoder written in C++ that was able to encode OBJ files in to the POP format and a WebGL component that implemented the decoding and the rendering of the geometry. The encoder had support for quantizing vertex positions and normals and we extended it to support the reading of PLY files that allowed it us to then bake also per-vertex color information which was very important for our scenes. We also modified the encoder so that it exports individual POP binary files for each level of detail instead of one binary file for all levels which further allowed us to optimize transmission rates by not having to transmit the whole geometry before decoding starts but instead doing transmission and decoding on a per-level basis. With this approach together with gzipping the POP binary files we managed to get the overall transmission size down from around 130Mb to 18 - 30Mb depending on the scene.
At this point we were quite happy with the results that we were getting and with the overall process but we were still working on an extended version of the initial raw WebGL implementation that we had discovered earlier. We really wanted to work with Three.js for all the extra functionality that you get by default when using a library like this so our next step was to develop a Three.js-based POP Buffer renderer.
The original WebGL implementation uses interleaved buffers for describing the memory layout and storing of the various vertex attributes ( position, normal, color ). Interleaving allows for the usage of a single Vertex Buffer Object ( VBO ) for all attributes instead of a separate VBO for each one and this technique is usually a good candidate for large static meshes. Looking at the implementation of interleaved buffers and how vertex attributes are setup inside Three.js we soon realized that we would have to make some modifications on that end in order for the rendering to work properly.
Usually vertex attributes such as positions and normals are represented by 32-bit floating-point numbers. This means that in order to store position and normal data for a single vertex you would normally require 12 bytes for position data and 12 bytes for normal data adding to a total of 24 bytes. The encoding algorithm packs the same data in a total of 8 bytes reducing the size by around 70%. It achieves this by performing a per-level quantization that translates position into a 6 byte representation ( 3 x 2 bytes ) and by oct-encoding of the normal data which results in a 2 byte representation. With the addition of un-compressed color information as a 4 byte representation ( 1 byte per channel / RGBA ), in the end, a single vertex can be represented by 12 bytes instead of 28 bytes achieving >50% reduction in size.
The main problem at that point was the way that Three.js sets up the vertex attributes, which assumes that the data type of the various interleaved attributes is the same as the type of the array used to hold the interleaved data. This is a fair assumption for most use cases but in our case we needed finer control over the description of the data since we had an array of 16-bit unsigned integers ( 2 bytes per array element ) holding attributes of different types/sizes ( i.e 2 bytes per component for position, 1 byte per component for normal / color ). We solved this issue by extending THREE.InterleavedBufferAttribute to accept a type argument together with the byte size of the attribute’s components and by modifying the Three.js WebGLRenderer to take into account those values for calculating stride and offsets when setting up the vertex attributes.
Once these modifications were in place we had a functional Three.js-based implementation for rendering POP encoded geometry which allowed us to then integrate, with minimal effort, additional functionality that comes with the library like camera and mouse interaction support, ray casting and picking used for activating POI’s, basic lights support and so on.
From that point on we were able to focus on the application logic and feel and also on the development and integration of the timeline component together with the POI’s implementation. The nature of the topic made it clear from the beginning that the look of the application had to be more diagrammatic than design oriented. We experimented with various levels of transparency on parts of the scenes and different color combinations in order to try and put the focus on the parts that were most important for the case.
The POI’s system is rendered through Three.js and works in parallel with the timeline. Its connected with the filtering mechanism and its purpose is to provide an entry point for additional info regarding events that occurred in specific locations inside each scene.
The timeline component offers a time-based representation of the events that took place that night. As in the case of POI’s, the user can filter events from the main application toolbar which are then further filtered by time and appear on the view sorted by different groups.
In addition to time-based filtering the timeline also acts as a selection tool for different snapshots in time of each scene. The user can switch between different timeframes of the night of the 26th of September, offering the chance to explore the events that took place in each scene over the course of the night.
The timeline was developed with D3.js and was integrated with the rest of the application as a React component ( same as for the Three.js scene ) since the main application backbone was based on React-Redux. This made it possible to easily connect the application modules that we were developing, together with the ones that the FA team was working on, by encapsulating different parts of the application in separate React components that shared only the necessary slices of the global state.
This project offers, hopefully, a glimpse on the possibilities of modern technology stacks and algorithms in developing support and enabling tools for civil society causes such as the Ayotzinapa case. Although this has been a ( mostly ) technical overview, at the core of the project remain the people, victims, their families and friends and all the questions that surround this case. We hope in the end they will find the peace and justice they are looking for. We would like to thank the FA team for engaging us with the project and all the open source communities out there for making it happen.