Governing Urban Artificial Intelligence
GovAI Coalition’s Vision for Responsible AI in the Public Sector
Artificial intelligence (AI) can transform the way public services are delivered. From generating real-time translations of public meetings to predicting traffic patterns for improved bus routes, AI tools have the potential to make government services more proactive, efficient, and accessible to all (Government AI Coalition | City of San José, n.d.). However, without addressing the concerns of privacy violation, safety threats, and racial discrimination, Generative AI (GenAI) tools can cause greater damage to public welfare when deployed in the public sector.
In the 18th episode of Urban AI Conversations, Emily Binet Royall, Smart City Coordinator at the City of San Antonio & Urban AI Member, hosts a conversation with panelists Albert Gehami, Privacy Officer for the City of San José, Jaime Wascalus, Chief Information Officer at the City of Saint Paul, Jonathan Behnke, Chief Information Officer at the City of San Diego, and Jiri Rutner, Chief of Business Operations, Behavioral Health Services, County of San Diego, to discuss GovAI Coalition’s vision for advancing responsible AI in the public sector.
Founded in November 2023, the GovAI Coalition is now a network of over 600 US-based public servants from more than 250 local, county, and state governments committed to the responsible and purposeful use of AI in the public sector. With members ranging from large cities like New York to small towns like Netherland, Colorado, the Coalition brings together a diverse group of government practitioners dedicated to tackling the challenges and opportunities presented by AI in the public domain.
Albert Gahami, Privacy Officer for the City of San José, shares the origin story of the Coalition. In the fall of 2023, the City of San José wanted to understand its procurement of AI systems better. After sending out requests for information to AI technology vendors, the city received a mixed range of responses. Smaller vendors tended to be more forthcoming, while more prominent vendors provided little information (Gahami, City of San José). This disparity raised concerns about using AI in public services without a baseline understanding of its capabilities and limitations.
Without a systemized way of addressing different aspects of AI procurement and knowledge sharing, city governments can struggle to keep pace with the rapidly evolving AI landscape and ensure the ethical use of these technologies.
“It’s one thing when Google uses some crazy AI stuff to predict the next time I wanna buy groceries on my phone, but it’s a whole other thing entirely when that AI is being used to determine how long I should be in prison or how long if I should see trial, or if I should be getting access to services,” Gahami said.
At the heart of GovAI’s mission is to ensure that AI technologies are implemented in a way that upholds ethical standards, protects privacy, and promotes equity and fairness in decision-making processes. The Coalition has created a suite of AI policy templates and knowledge-sharing tools that outline ethical principles and guidelines government agencies should follow when deploying AI technologies.
The conversation emphasizes four main components of the GovAI Coalition’s practical AI governance tools.
- Its ability to enable Use Case Coordination. The Coalition facilitates sharing AI use cases and lessons learned across its member agencies. By coordinating the exploration of emerging AI applications, such as GenAI and computer vision, the Coalition helps public entities learn from each other’s experiences and avoid duplicating efforts.
- The development of a Vendor Registry, a database of AI FactSheets. Recognizing the importance of working with ethical and responsible AI vendors, the Coalition built a database of companies that align with its values and principles. This registry aims to guide public agencies in their procurement decisions and encourage AI providers to prioritize transparency and accountability.
- The group has developed template contract language that public agencies can use when purchasing AI-powered products and services. This legal framework helps inscribe the Coalition’s values into binding Vendor Agreements.
- Acknowledging that things can go wrong with AI systems, the Coalition has also created an AI Incident Response Plan to guide public agencies on how to address issues, such as algorithmic bias or privacy breaches, should they arise. All of these resources are public on the Coalition’s website, and they are extending their invitation for more public agencies to join.
After introducing the work of the Coalition, the panelists shared their insights on some key challenges and opportunities to governing urban artificial intelligence.
Jonathan Behnke, Chief Information Officer at the City of San Diego, shared their city’s pilot project in 2023. The city trained a GenAI chatbot on its intranet’s top 100 searches and pages. The project used three leading AI solutions — Google Gemini, Azure OpenAI, and Amazon Solution — to assess performance, accuracy, cost, administration, ease of use, and user experience. After overcoming the initial hurdle of training staff to operationalize the new technologies, Behnke emphasized that the AI model excelled at summarizing large documents and providing quick access to information, offering significant time-saving benefits.
“I was super impressed at how the GenAI solutions captured information from some of our large documents and summarized the main pain points. But we also saw that poorly written documents and some out-of-date content were obstacles to getting good results. And I think for us, that helps drive our city departments to keep their content more up-to-date,” Behnke said.
Many other opportunities are identified. AI-powered chatbots and virtual assistants can streamline access to government information and services, providing residents with 24/7 support and reducing the burden on call centers and administrative staff. AI tools can support accessibility and language translation of government websites and documents. AI tools can also help to automate digitizing old records and the associated metadata.
Besides the concerns of misinformation, privacy and automation bias, Behnke explains that another challenge directly experienced by their pilot project is a way to evaluate AI performance and effectiveness. Traditional IT testing methods are not suitable for GenAI, as user interactions can vary significantly and responses may be subjective. To address this challenge, the city of San Diego added user sentiment analysis to gather feedback on the tool’s performance, and focus on making continuous improvements.
Furthermore, obtaining precise information from AI vendors about their algorithms, data models, and testing procedures is difficult. Much of this information is considered proprietary, which companies do not want to share.
Therefore, clear policies and standards regarding data ownership are needed. In the case of the city of San Antonio, a strict requirement is in place stating that the city owns the data generated by the technologies it uses and purchases. This policy is driven by the belief that data generated by AI technologies are public assets and should be accessible to the public.
Additionally, the city has an “open-by-default” policy, making any data subject to open records requests. However, there is also a challenge to balance the need for open data with the protection of sensitive information that AI technologies generate.
The last challenge is being able to better understand and prepare for the local impacts of AI on jobs, which is already happening in some sectors. Jiri Rutner and Jonathan Behnke from the City of San Diego explained that due to less attractive compensation, city governments can struggle to hire and retain specialized staff. Focusing on three areas — compensation, training, and local job impact — will significantly benefit agencies adapting to and leveraging AI. Organizations should prioritize upskilling and training their staff, leveraging small pilot projects to secure wins and increase enthusiasm for AI technology in local government.
The challenges and opportunities discussed in the conversation provide a solid grounding for evaluating the emerging occurrences of AI vendor contracts in the public sector. The California Department of Transportation (Caltrans) just awarded the first GenAI vendor contracts in California history in May 2024. The program aims to “test innovative solutions to protect vulnerable road users and improve overall traffic management”. The news follows California’s new set of guidelines on GenAI released in March 2024.
All eyes are on city governments and the public sector across the globe as they grapple with governing urban AI effectively and responsibly.
The Coalition presents a collaboration framework centering both policy and operation, so that information about best practices and lessons learned are shared across agencies on a national scale. The path forward should be rooted in trust, accountability, and shared responsibility.
References
Associated Press. (2024, April 3). NYC’s AI Chatbot Was Caught Telling Businesses to Break the Law. the City Isn’t Taking It Down. US News. https://www.usnews.com/news/us/articles/2024-04-03/nycs-ai-chatbot-was-caught-telling-businesses-to-break-the-law-the-city-isnt-taking-it-down
California Department of Technology. (n.d.). GenAI Executive Order — CDT. California Department of Technology. Retrieved June 15, 2024, from https://cdt.ca.gov/technology-innovation/artificial-intelligence-community/genai-executive-order/
Executive Department: State of California. (2023, 9 6). Executive Order N-12–23. Governor of California. Retrieved June 15, 2024, from https://www.gov.ca.gov/wp-content/uploads/2023/09/AI-EO-No.12-_-GGN-Signed.pdf
Government AI Coalition | City of San José. (n.d.). Government AI Coalition | City of San José. City of San Jose. Retrieved June 13, 2024, from https://www.sanjoseca.gov/your-government/departments-offices/information-technology/ai-reviews-algorithm-register/govai-coalition
IBM Data and AI Team. (2023, October 16). Shedding light on AI bias with real world examples. IBM. Retrieved June 13, 2024, from https://www.ibm.com/blog/shedding-light-on-ai-bias-with-real-world-examples/
Johnson, K. (2024, February 8). California plans to use AI to answer your tax questions. CalMatters. Retrieved June 15, 2024, from https://calmatters.org/economy/technology/2024/02/cdtfa-generative-ai/
Johnson, K. (2024, March 21). California issues guidelines for government AI buys. CalMatters. Retrieved June 15, 2024, from https://calmatters.org/economy/technology/2024/03/california-ai-purchasing-guidelines/
Urban AI. (2024, May 6). Urban AI Conversation #18 — Governing Urban Artificial Intelligence. YouTube. Retrieved June 13, 2024, from https://youtu.be/MuPkWq07H2w
