ArcGIS Online — Patterns to work with developers

ArcGIS Online architecture pattern for dummies

ArcGIS Online is a platform that offers a plethora of out-of-the-box options. But there are few cases where you need to run your own code in the cloud, with ArcGIS Online. Here is what you need to know about patterns and solutions for custom code with ArcGIS Online. We will also mention the business cases they serve. Tag along, there is no need to know anything about programming.

ArcGIS Jupiter Notebooks

ArcGIS Jupiter Notebooks was my favorite improvement to the platform in 2020, for a good reason. It lets you run your own Python code on ArcGIS Online. It is a good option to automate and schedule:

1. Your ETL workflows. If you need to fetch some data from an external data source, process it, and update a feature service, then Notebooks for ArcGIS can be the perfect tool.
2. Your administrative workflows. Do you want to regularly scan for users who have not logged in, content dependencies, generate custom reports? Then Notebooks for ArcGIS can be the perfect tool.
3. Your data analysis workflow. Do you need to regularly inspect data, generate graphs, and take automated decisions based on it? Then Notebooks for ArcGIS can be the perfect tool.

There are many reasons I love Python notebooks. It ranges from the fact that some notebook environments do not consume credits to the fact that they come with task schedulers. In order to know more, check my article about my 4 reasons to use notebooks for ArcGIS Online.

The main reason though, is that you don’t need anything else than ArcGIS Online. This is the simplest scenario of custom code in ArcGIS Online.

Hosting Custom Code

Here I will speak mostly about my experience with Azure. However, I am making the assumption that this should be similar if you’re using Amazon as a code hosting platform. There are typically two workflows for which I do custom code with ArcGIS Online or Portal for ArcGIS.

1. Custom Application Templates

In some cases, you might need to create a new groundbreaking app template, and expose it on ArcGIS Online. Of course, you want that app to become a configurable template your users can create applications from. Using Azure, it is possible to host your custom application template on Azure and register it on ArcGIS Online to make it available to your users.

Creating a new JavaScript application in ArcGIS Online using a custom template

Custom configurable application templates can be created for both versions of the Esri JavaScript API (3 and 4). Esri provides boilerplate code to create a configurable application template using the API version 4 on their Github repository here. This type of custom code is supported by both ArcGIS Online, and Portal for ArcGIS.

2. Custom Web App Builder Widgets

The second scenario is hosting custom Web App Builder widgets. This is the scenario when none of the Web App Builder widgets meet your needs, and you need to create your own. Your custom widget can then be registered and used in Web App Builder on Portal for ArcGIS.

The workflow is quite simple. You create a widget in Web App Builder developer edition. Then you test it in an app. Once it is done, you can copy it to a web server, and register it on Portal. People can then use your widget in Web App Builder applications on Portal. The workflow is documented here.

As a developer, you focus on only coding the missing bit in your apps. Development is costly and triggers maintenance costs, therefore it needs to be minimized. As an application creator, you can then create Web App Builder applications that use those custom widgets without knowing anything about code.

App creator adding a custom widget in Web App Builder

Unfortunately, custom widgets are supported in Portal for ArcGIS, but not on ArcGIS Online.

In both cases, your custom code can be hosted in several ways. You can use your public webserver if you have one. Azure Blob storage is also a good fit if your environment can access the internet. You will need to update the security settings of ArcGIS Online or Portal to allow CORS requests to your custom code.

Webhooks

Webhooks are a mechanism to call a REST endpoint when an event is triggered. If your organization needs to know when new content is published on Portal for ArcGIS. You can do it two ways:

1. You can query the content of your Portal every 5 minutes, and see if any new item has been added since the last time you queried the content. This is the inefficient way: you are wasting resources on the machine that queries the portal, and the portal itself.
2. You can ask Portal to notify you when a new item is published. When it happens, Portal will call a URL you have configured, and give the information about what new item was just added. This is a webhook, and this is the best way to tackle that problem.

The URL you’re configuring in your webhook can point to any of your custom applications. That application can send emails under certain conditions, create tasks in outlook, update a website. With this, the world is your oyster. I love using webhooks with Azure Logic Apps.

Beware the webhooks work differently on ArcGIS Online and ArcGIS Enterprise. As of October 2021, ArcGIS Enterprise supports events associated with groups, users, and items. ArcGIS Online supports events associated with Feature Services (update, insert, delete).

Here is an example of a useful webhook. An organization wants to notify reviewers when an item is shared with a group for review. An Azure Logic App is registered with the item/sharing event in Portal. The Azure web app checks the sharing properties contains the identifier of the review group and notifies reviewers they have some work.

An example of architecture using Webhooks

This example saves the reviewer the need to check daily for new content shared with the reviewer group.

Koop

Imagine a third party provides data you need to use in ArcGIS online. It is not in a GIS format that ArcGIS Online knows, and you don’t want to fetch the data to host it yourself. This is when Koop kicks in.

Koops lets you take data hosted on the web in different formats (CSV, Craigslist, Google Analytics, Elastic Search), and make them look like feature service on the fly. If the format is not supported, you can write your own Koop connector.

The workflow is that Koops read your input file, and exposes a Feature Service URL that can be queried like any regular ArcGIS Online feature service. This is ideal if you need to access a massive amount of data from a third party, especially if that data is updated frequently.

Let’s take an example with a simple CSV. I want to be able to visualize a CSV that contains hydrometric stations in a CSV file as a feature service. I can install Koop on my machine. I can ask Koop to read the CSV file from a URL and display it as a feature service.

The list of water stations is available here as CSV. It looks like that:

NRCan hydrographic data as an online CSV

If I read it through my Koop service, it reads like a feature service that I can use in ArcGIS Online maps.

NRCan CSV exposed dynamically as a feature service through Koop

The demo Koop code is available on my GitHub.

This is it for this series on custom solutions with ArcGIS Online and ArcGIS Enterprise. This idea is once you know them, can choose the pattern that meets your needs and focus on the solution.

Of course, there are more complex patterns you can use with Docker, Microservices, etc, but this kind of complex design should be avoided if you can use one of the patterns above. They are harder to maintain. What’s the point of working hard, if you don’t work smart?

As usual feel free to comment for ideas, like if you enjoyed the read, and follow me if you want to learn more.