Building a Public API for the Spatial Equity Data Tool
The Urban Institute’s Spatial Equity Data Tool (SEDT) enables local, state, and federal leaders to upload data and quickly assess whether place-based programs and resources — such as parks, libraries, Wi-Fi hotspots, or electric vehicle charging stations — are equitably distributed across neighborhoods and demographic groups. To date, the tool, which was launched in 2020 and expanded in 2021, has been used by a variety of experts and nonexperts alike. For example, Bloomington, Indiana’s Bicycle and Pedestrian Safety Commission analyzed sidewalk funding allocations, and advocates in Cincinnati identified economic and racial inequities in car crash locations.
Through continual engagements with local government and nonprofit users, we identified two key barriers keeping organizations from more fully incorporating the tool into their decisionmaking and processes. First, instead of using the point-and-click interface on Urban’s website to upload a dataset whenever new analysis was needed, organizations wanted to embed the SEDT within their own tools and data workflows to make it a core part of institutional processes. Second, organizations requested technical assistance to identify and maximize opportunities to transform decisionmaking and resource distribution processes with the SEDT.
In short, while the user-friendly point-and-click web tool is great for many users, we needed another solution to support organizational power users. What we needed was a public application programming interface (API), which we are excited to release today.
What does the public API do?
The public API enables users to interact with the SEDT programmatically. Users can submit code for an analysis request along with their data to the API and receive a structured response with the results. Users can then automate the process of submitting multiple analyses to the SEDT and incorporating results into organizational processes and tools, such as dashboards, data catalogs, and custom data tools. This automation takes an otherwise time-intensive manual process and completes it in a few seconds, providing the user far greater quality control, customizability, and time efficiency. The Urban Institute has had great success using APIs to make our tools and data more accessible, including through our Education Data Portal API.
The SEDT API also gave us the opportunity to add advanced functionality requested by our power users without compromising the streamlined user-friendliness of the web tool. The API introduces three new pieces of functionality that aren’t available in the web tool:
· Users can choose which year of American Community Survey data to compare their user-uploaded point data (which we call a resource dataset) against to calculate the disparity scores.
· Users can upload supplemental demographic datasets that contain information on additional demographic groups beyond the web tool’s built-in groups to include in the calculation of demographic disparity scores.
· Users can upload supplemental geographic datasets that contain information on additional target populations beyond the web tool’s built-in geographic baselines to use in the calculation of geographic disparity scores.
These new features significantly expand the applications for the SEDT. If a government agency wanted to measure equitable targeting of a specific program, for example, they could upload custom demographic and geographic data on that program’s specific eligible population that may not be one of the built-in datasets for the web tool (e.g., small businesses for a small business grant program) for more accurate analysis.
How can people use the API?
An illustrative example of how people can use the new API comes from our partners in the Office of the Los Angeles City Administrative Officer, who wanted to use the SEDT API to center equity in the city’s budget process.
Together, we developed the Measure of Access, Disparity, and Equity (MADE) tool, which leverages the API to analyze how the city allocates its resources, helping leaders better understand disparities in communities’ access to amenities. MADE integrates the disparity scores produced by the SEDT with the city’s new equity index in a custom City of LA dashboard. Departments can use this dashboard to develop annual budget proposals that more equitably target resources across neighborhoods and demographic groups. The city’s Bureau of Street Lighting, for example, tested the tool to develop their 2023 budget proposal for new electric charging stations. Integrating the SEDT directly into city processes through the API removes barriers to use by city staff. The MADE tool was shared in the official budget guidance received by all departments, and 300 city officials charged with developing budgets for 40-plus city departments attended trainings on the tool.
Through testing the prelaunch API with LA and our other technical assistance partners, we received invaluable feedback on how to improve our API documentation and overall user experience (not to mention caught and resolved several edge cases!). We’re grateful to all our partners for their time and input that made the API better.
How does the public API work?
The public API — just like the private API that currently powers the web tool — uses the Django REST framework as its foundation to interact with the tool’s analytical back end in a serverless AWS Lambda architecture. As a previous Data@Urban blog post outlines, Lambda’s event-driven, serverless cloud computing service is used across Urban for highly scalable and cost-effective applications. To address the growing complexity of the SEDT tool, we integrated an AWS Step Function into our software system. AWS Step Functions are designed to orchestrate and coordinate workflows by governing the sequence of steps that occur in the AWS backend after a user submits a request to the public API, as shown in the diagram below.
After a user submits an analysis request to the API using the /upload-user-file/ endpoint, a series of lambda functions manage the analysis pipeline, where the output of one function forms the input of the next function. As the public API allows users to upload optional demographic and geographic files, the number of files required for the equity analysis and the amount of time it may take for those files to upload to S3 can vary across requests. To manage this uncertainty, the first lambda function determines which files a user will upload based on the API request parameters that the Django API saves to S3. It then passes this information to the second lambda function, which repeatedly checks S3 to determine if the required files have finished uploading. If not, the step function pauses the workflow, then triggers the second lambda function again to restart the check. If yes, the second lambda function triggers the third and final lambda function, which performs the equity analysis and writes the results to S3.
The step function allows us to orchestrate these individual interconnected AWS services into a single workflow. It also offers a useful visual interface for the workflow that identifies at which part of the process an error occurs if the workflow fails, making testing and debugging significantly easier. The step function also provides a flexible platform for future development. Past updates to the tool have expanded the workhouse equity analysis lambda function. However, as SEDT software and capabilities expand, we can easily add new functionality to the step function pipeline through smaller, modular lambda functions without overcomplicating our existing functions.
After the user submits their analytical request to the /upload-user-file/ endpoint, the API response returns a unique analysis ID that they can use as a parameter to the public API’s other two endpoints:
· /get-output-data-status/{id}/ takes the unique analysis ID as a request parameter and returns a JSON response with updates on the analysis progress and any warning or error messages.
· /get-output-data/{id}/ takes the same unique ID as a request parameter and provides a response with the analysis metadata as well as the geographic and demographic bias analysis results.
How can I use the API?
Our API documentation outlines how to use the API, including providing definitions of the request and response parameters for each endpoint and sample API calls in R and Python programming languages. The documentation also outlines how to use the new analytical features available in the API, such as uploading custom demographic and geographic datasets.
We have also released the sedtR R package to provide a convenient interface to the API for the R programming language. The package includes functions for interacting with each endpoint and a single wrapper function call_sedt_api(), shown below, that manages the analysis from start to end, pinging each endpoint as needed.
The R package also includes some useful functions for analyzing results, including functions to produce a similar demographic disparity chart and geographic disparity map to the web tool.
What’s next?
We plan to continue updating the SEDT API and the sedtR R package and hope to release a corresponding Python package. We are actively seeking new technical assistance partners with organizations incorporating the API into their workflows. We’d love to hear from you about how you’re using the API, if you’re interested in partnering for a larger-scale integration, and if you have any feature requests. Please don’t hesitate to reach out to us via GitHub issues, or by email at sedt@urban.org.
