Serverless ArcGIS Feature Service Automation and IaC

An introduction to a series on managing data hosted in Esri’s ArcGIS portals programmatically.

Infrastructure as code (IaC) and serverless automation have been booming in recent years, however their application to web based GIS seemed limited until recently. Technologies are converging to make cloud based GIS data management simpler and cheaper than ever before.

A common workflow to publish and update data in Esri’s ArcGIS Online and ArcGIS Enterprise portals is publishing the data from a staged MXD file in ArcGIS Desktop, and more recently running a similar workflow from an APRX file in ArcGIS Pro. These workflows use Esri’s service definition files generated from existing data on disk to efficiently update the data in the portal. But what if you have a free and open API you’d like to work with in your cloud GIS that is neither in Esri REST format or OGC compliant? Previously you’d have to pull the data down to a local machine, process it into an Esri compatible format, and publish by one of the two workflows mentioned above.

A few years ago though, Esri implemented the ability to directly upload flat files like CSV and GeoJSON and automatically generate feature services from them in ArcGIS Online on your user’s content page.

“Okay, excellent! But I thought this article would be more about IaC and serverless?” — The thoughts of the person reading this article

I’m getting there! Esri’s ArcGIS API for Python brings (most) of the functionality of their portal GUI to Python. This opens up a host of possibilities, particularly when combined with the ability to package up Python libraries for use in AWS Lambda. See where we’re headed now?

This workflow won’t be the answer for every use case and it has several inherent limits based on the technologies we’re using:

• Lambda’s temporary workspace is limited to 512 MB, meaning our intermediate file for upload to AGOL cannot exceed that size
• Lambda function timeouts are limited to a 15 minute maximum, so our function can’t take any longer than that
• We can’t do heavy geoprocessing when publishing and updating data from the API to AGOL

Still interested in learning this workflow? In the following three exercises we’ll cover:

Exercise 1: Automating deploying and updating a feature service with the ArcGIS Python API

Exercise 2: Creating an AWS Lambda to update your service on a schedule

Exercise 3: Using Troposphere to generate a Cloudformation template for deploying infrastructure as code and wrap up

Fair warning, this isn’t for the faint of heart and you’ll benefit from a familiarity with the basics of Linux, AWS, and Python, as well as using a command line interface (CLI). Because almost all Esri users are on Windows, I’m assuming that’s your OS, sorry Linux and Mac users, but it is what it is.

Setup

Software

• Python 3 (3.6 or above)
• WinSCP
• Putty
• PyCharm (optional, really any Python IDE will work here)
• AWS CLI (optional)

Licensing

• ArcGIS Online or ArcGIS Enterprise Publisher level user
• AWS user with privileges to use IAM, Lambda, EventBridge, Cloudformation, S3, and EC2 (programmatic access optional)

Download or clone the sample code from the GitHub repository.

Using Command Prompt (or your CLI of choice) navigate to the directory you’ll be working in and create a virtual environment, using the provided requirements.txt file to install the needed python libraries.

cd C:\my\working\directory
python -m venv medium-post
.\medium-post\Scripts\activate
pip install -r requirements.txt

Once you’ve successfully setup your environment, you can begin Exercise 1!

Exercise 1 →