Automation is a ladder: part one
“Automation is a ladder. Many who try to climb it fail and never get to try again.”
Well actually, it’s not quite that severe (and my apologies to GoT’s screenwriters for the misquotation). However, the tension between learning to automate your work and delivering against immediate deadlines is a familiar one for many of us.
How do we balance the long term benefits of increased efficiency against the immediate cost of learning a new skill? As a species, prioritising long-term outcomes is not our strong suit, and we in the geo community are no exception.
In the Esri ecosystem, our dilemma is compounded by the range of options available for automating things. In ArcGIS Pro, these include:
- An internal Python console
- Model Builder, a graphical automation framework
- Script Tools, which attach custom Python scripts to Pro’s user interface
- Tasks, a new-ish framework for assembling and documenting multi-stage processes
- ArcPy, a Python library for automating desktop tasks that can be invoked from any Python environment
- The ArcGIS API for Python, another Python library for managing web maps, web apps and other online content
Are you confused yet?
If we are going to devote precious time to improving our craft, how can we be sure that we will get the best bang for our buck? Which tool should we prioritise? It’s easy to feel overwhelmed and perhaps frustrated by such a range of choices, even if this flexibility is ultimately beneficial.
How we approach this decision depends on our situation. Existing knowledge, access to special resources (like helpful colleagues), or legacy processes that you work with all might influence your choice.
There are also strengths and weaknesses of each tool that don’t become apparent until you have some hands-on experience. That’s where I hope this series will prove useful.
The first bit of good news is that the ArcGIS Pro team have built in a couple of tricks that anyone can benefit from straight away. You might use these already, but it’s perfectly possible to work with the software for a considerable time without uncovering these features.
Analysis history
Every Pro project includes a full history of the geoprocessing (GP) tools that you’ve run. All analysis and almost every data interaction in Pro takes place through this GP framework, either overtly or covertly. This makes the project history a valuable document of which parameters you used for each job, and of what got saved where.
Need to re-run something that you did last week, or even last year? Find the history item and just press go. Reusing the same process in a new project? Just feed a new data set in and keep the other parameters the same. As of Pro 2.5, you can also schedule history items to run again from inside the app.
Drag-and-drop
It’s easy to drag and drop data onto your map, but you can also drag GP tools from the Search pane, or result items from your analysis history, into Pro’s built-in Python console.
This helps to cut out a large part of the learning curve for Python. In fact, you are not far from saving yourself a lot of time already!
I used both of these features in a recent personal project, a map of Ronnie Scott’s and Soho based on John Nelson’s cross-stitch map style (because, why not).
This included OpenStreetMap data from Geofabrik. For the sake of efficiency I wanted to extract the relevant part of eighteen national shapefiles (streets, buildings, points of interest, etc). Without automating these individual operations, this could take up a significant chunk of a day between waiting, checking back, and setting things off again. Manual error is the source of the vast majority of technical problems, so let’s see how to save time and assure quality results with some Python.
You can copy the code below directly into Pro’s Python console, and iterate through each item in a folder or geodatabase, with only three simple changes required:
Having run the Clip tool once, and by dragging the GP record in from our project history, we get some fully-formed Python syntax that will achieve the same result for us in a script.
All that we need to do now is to substitute our output variable name (out_name) to create a uniquely-named extract from every shapefile in the folder, whether there are two, or two thousand:
Hit return and soon enough, everything is ready to go! You’ll even see detailed results from each execution of the Clip tool in your analysis history.
I hope you’ve found this useful. Perhaps you’re even inspired learn a bit more about Python. Please share, let me know your thoughts, and stay tuned for part two.
