Going Global — Part I
Feature requests are one of the most difficult parts of software development. The majority of the time they come from non-developers with no conception of the time or effort required for implementation, nor with a good understanding of the developer’s available time and priorities. To be fair, it is also often difficult for a developer to make these same estimates and calculations, leading to uncertainty on both sides that can result in the original request being too ambiguous to act on.
Anecdotal evidence can be dangerous when used to prioritize development time, as the denominator is never known — there may be five people loudly complaining, but hundreds or thousands that don’t care (or hundreds or thousands that do care but haven’t said anything). Because of this, it is often a wise investment of time to investigate the relative importance of a feature before resources are allocated to its development.
For the past few months, the API that I wrote for macrostrat.org has been powering Flyover Country, an iOS and Android app that allows users to learn about the natural features they are flying over, including bedrock geology. It was released in December 2015, but in the past month or two it has really started to gain traction.
A frequently requested feature from the Flyover Country team is the ability to query our global geologic maps. While this sounds straightforward, there are many complicating factors, including:
- The global maps (internally referred to as `tiny`) are very coarse in resolution compared to the maps currently used
- Ideally, the global maps would simply fill in the gaps between higher resolution maps. This would require a custom geometry to be created that would need to be separately maintained and updated as new maps are added
- The above solution is potentially a very expensive (i.e. time and resource-consuming) operation
- Global flight paths can be very long , which results in longer processing times to create a response, as well as a potentially massive Geo/TopoJSON object (tens of megabytes) to send to the application.
With these challenges known, and a mind toward allocating resources in the direction of highest impact, I set out to quantify the desire for these global maps by creating a map of the requests from Flyover Country to the Macrostrat API.
The first step is to extract requested flight paths from our server logs. When using Flyover Country, the user clicks on a map to indicate a start and end point of their journey, from which a linestring is generated, buffered by about 200km, converted to WKT, and sent as a query parameter to our API, meaning each polygon shows up in our logs.
It is easy enough to extract the polygons from the log file and insert them into a PostGIS database, but the slightly more complicated part is deriving an original start and end point from the polygon representing a buffered linestring. To do so, the minimum and maximum longitude of each polygon must be found— this gives you the minimum and maximum coordinates of the polygon. Using that coordinate pair, a great circle is drawn between them, and PostGIS’s ST_LocateAlong function is used to find the coordinates of a point 200km from the original minimum and maximum coordinates. This gives us our estimated original start and end points, which can then used to draw the original paths.
With some rough geometry in hand, it was time to make a map and see if there were any clear trends in the use of Flyover Country, and what the potential impact of global geologic map coverage could be.
Some clear trends immediately emerge once visualized: the majority of the traffic is clearly focused on the northern hemisphere, specifically the USA and Europe (if you look at a map of daily global flight traffic, similar patterns are apparent). While it is nearly impossible to tease out domestic US and EU traffic from this map, it is easy to see domestic traffic in South Africa and Australia, as well as a significant amount of traffic between the continental US, Hawaii, Alaska, and the Caribbean.
Perhaps the biggest surprise, and most important take-away, is that Flyover Country users are traveling outside the continental US, the primary area currently covered by our geologic maps. Flyover Country went live in December 2015, and in that time over 55,000 geologic map requests have been made to Macrostrat, 29,000 of which have start and end points in or near the continental US, representing only slightly more than half of total traffic.
With these lessons learned, it becomes much easier to design services that best meet the needs of users, which in this case is enhanced coverage for geologic maps. Additionally, these maps allow us to see usage trends that were not previously visible. For example, the maps indicate that there is a significant amount of traffic between a handful of cities in the continental US (primarily hubs for major carriers), which might enable us to cache common routes for faster response times.
The next step is to act on these insights, and design a data layer and corresponding service that allows for quick access to this data, which will be covered in Part II.
