Making ArcGIS Server Screaming Fast.
It shouldn’t come as a surprise that the most popular GIS server across the Government, Forestry and Enterprise sector is ESRI’s ArcGIS Server. As a developer, I enjoy working with Open Source geospatial tools far more, but for some clients they are simply not an option. I have been building web apps that consume ArcGIS Server Services for about 4 years now. In each project, I have had to take into account the relatively bad performance and reliability of ArcGIS Server. For example on one past project I discovered that harvesting metadata from the REST services on request actually took down the services under load so I was forced to create a work around and rate limit the harvest and store a static copy of the metadata locally with the app.
Another challenge is that the Javascript SDK ESRI ships is built on top of Dojo. Dojo uses the Asynchronous Module Loading Definition (AMD) as a module loader which doesn’t play nicely with modern bundlers like Browserify or Webpack. Modern bundlers use either CommonJS (NodeJS) or Universal Module Definition (UMD). Its for this reason that in most projects we use either LeafletJS or the ESRI’s leaflet extension esri-leaflet as our map client when we have the choice.
Server challenges aside, more recently I have been working on an exciting project for the Alberta Biodiversity Monitoring Institute (ABMI). The ABMI has developed an impressive data inventory which quantifies human footprint, multiple species intactness, richness, and uniqueness as well as individual species maps for thousands of birds, mammals, vascular plants, old forest birds, and non-native vascular plants. The data was aggregated to a 1 km^2 grid with cells covering the province of Alberta.
In total there were roughly 2000 datasets we needed to expose, we knew this was going to be a challenge with ArcGIS Server. We have previously run into issues with ArcGIS Server having either too many services or too many layers in a service. We brought a proposed approach to ESRI Support and the support representative cautiously agreed that setting the default visibility of all layers to false might enable us to publish all these layers though he wouldn’t really recommend it. Not having a lot of other options in this case we pressed onwards.
Client-side caching, Map Tiles to the Rescue.
For our first iteration we used the Dynamic Map Layer from esri-leaflet which was painfully slow to render (30–40 seconds). The Dynamic Map Layer does a good job at slicing up the images requested but their bounds are screen dependent and as such don’t leverage browser caching. On every pan/zoom the map would request an entirely new image from the server which was slow and created a very poor user experience even on return visits.
Slippy maps and map tiles have been around for a while now and solve this browser caching problem. ArcGIS Server comes with a few options for tiling but we decided to go with the WMS service and the LeafletJS TileLayer.WMS layer. The ArcGIS Server dynamic caching option was considered but since we wanted to decouple the cache layer from the ArcGIS Server it was quickly ruled out. Since our app allowed the user to filter data by passing in layer definitions the permutations were far too great and would quickly overwhelm the ArcGIS Server. One positive thing is that ESRI extended the built-in WMS service to allow layer definitions which we needed for this project to enable the user to filter out the values they want mapped. (i.e. show me all human footprint where values are greater than 60%.)
Server-side Caching, AWS CloudFront to the Rescue.
In a perfect world there would be a 1:1 ratio between services and CPU cores (or 2N+1 as is recommended for AWS deployments). In order to stretch the instance / pooling model to accomodate the thousands of layers we were up against we changed the default minimum instances per service from 2 to 0. This causes each service to take more time to initialize after a request comes in but it is less likely to cause resource constraints and crashing. Tiling worked quite well and now map tiles were being cached by the browser but the initial tile rendering was taking a while to render because of this service initialization.
We pondered on what would be the easiest strategy we could use to add a caching layer between the user and server. We thought of reverse proxies like NGINX or even a simple NodeJS proxy app.
We leverage the cloud extensively at Tesera and specifically Amazon Web Services (AWS). AWS offers a Content Delivery Network (CDN) service called CloudFront which offers caching among other features but we had never used it with a non-AWS resource before. Creating a CloudFront Distribution is quite easy. You create a it, configure a few options, point it to an AWS resource or simple URL and boom you have a very fast almost zero maintenance web cache. A CloudFront Distribution can be placed in front of most http resources. We wondered if we could put it in from of ArcGIS Server and what the results would be. We went ahead with the experiment and the results were impressive! Tile load times went from 20–30 seconds to a few hundred milliseconds. The app felt way more snappy, supporting the relatively modern user interface much better which made us and ABMI very happy.
Icing on the cake, Priming the Cache.
The only remaining problem was that there was still that initial hit when the tile was first requested. To solve this, we wrote a script that calls our rasterize/print service (future blog post) with the stateful url of the app including the views we wanted to cache. The rasterize service is a simple AWS Lambda function exposed via the API Gateway. The Lambda function utilizes PhantomJS and its example rasterize script exposed as a service. It essentially takes a screenshot representing the app state and returns an image. We scripted to load zoom levels from 5 to 8 of the extent of Alberta, about 16000 request in total. I rate limited the script to about 3 per 30 seconds to avoid sending the ArcGIS Server into a frenzy.
Although making ArcGIS Server itself faster might be an impossible task, we leveraged AWS CloudFront to help us make it feel faster. I hope this article was informative and helps you make the best of your next ArcGIS project. If you have had similar experiences and would like to share them ping me on Twitter. Yves Richard
