Towards Rules for Automation in Government

Caveat: This is a personal view of work underway that I’m leading. What I describe is subject to incredible change as this policy work winds its way through government and consultations. Our approach may change for reasons that I’m simply not privy to, and that’s fine. This is meant to solicit ideas, but also show the complexity about what it takes to make policy. I hope that people find it useful, particularly students of public admin. It also represents my view of the world only, and neither my organization’s or the Government of Canada writ large.

“Le Parlement, effet de brouillard” — Claude Monet, 1903. The best symbol for rule-making in a time of disruption.

In my humble opinion, I have one of the greatest jobs in the country right now. A typical day is often chaotic, but at its heart, I get to be the pen on the rules that will govern how federal institutions can use AI and other forms of automation. To some (well, most) “designing rules” perhaps isn’t as interesting as an actual AI project, but as someone with years of experience working in science policy and ethics, this represents one of the most challenging tasks of my career. I fervently believe that innovation happens closest to the user; as I sit in an ivory tower in a galaxy far, far away, I feel like my primary role is to facilitate and shape that innovation to be as productive and inclusive as possible. Alistair Croll talks about a “bias for action” while accepting, understanding, and managing externalities. What I’m trying to do is the latter part so that federal institutions can get to the former with less uncertainty.

AI is a rapidly evolving space, and trying to create rules in a time of disruption is risky. Too severe and innovation can be hindered; this is unacceptable during a time when the Government of Canada is embracing digital culture. On the other hand, if the rules don’t have meaning and teeth, and Canadians will not be sufficiently protected from the negative outcomes of this technology, like this or this. Trying to strike the right balance between facilitating innovation while being protective of right is a challenge, and one that benefits from ongoing discussions with different sectors across the country. It also means that I might work hard to build a consensus around a set of rules that we try out and have to scrap and redesign after a year in deployment because they don’t work.

Why rules around AI? To answer that question, I urge you to read the AI Now Institute’s 2017 report, as it expands on the reasoning much more elegantly than I ever could. Delegating judgement in matters of government previously reserved by humans to machines is, in my opinion, worthy of rule-making. In short:

  • All automated systems, whether they use decision trees or deep learning or another methodology, bring with them an issue of scale. Simply, if a human makes a series of errors, the impact of their errors is limited to those affected by the decisions that cross their desk. An employee silently harbouring racist biases limits the impact of their bias to their delegated decisions. An automated system making biased decisions, or decisions that reinforce inequalities, affects an entire population.
  • It’s critical that procedural fairness be maintained in an era where machine decisions play a growing role in the lives of Canadians. The government should be able to explain decisions about people, do so clearly, and with enough information that the decision can be challenged. Our systems don’t only have to work well; the population has to trust them.

My management gave me a pretty wide berth in this area (thanks!). So where did I start? Obviously with toughest possible starting point! I have proposed the development of a “Treasury Board Standard on Automated Decision Support Systems,” (title TBD) a binding policy instrument that will provide federal departments and agencies with a flexible, technology-agnostic set of rules designed to help them innovate responsibly with minimal paper burden. This policy would guide federal entities only, not the wider digital economy in Canada. It’s an important distinction because most of the media requests I get ask about the broader picture, which is outside my mandate.

Automated Decision Support Systems

Automated systems that make recommendations or decisions about individuals have been in use for decades. Banking and insurance were the vanguard industries in these areas, and it is the concept of automated decision making that underpins, for example, each credit card transaction you make. The introduction of machine learning may have changed the way that the decision is made, but it does not take away from the fact that machines have been making lots of decisions for a long time about people and society seems largely comfortable with this idea.

An engraving of Lloyd’s of London Subscription Room in 1809. The insurance market was one of the first examples of mathematical tables being used to assess risk of human activity to reduce subjectivity. By Thomas Rowlandson (1756–1827) and Augustus Charles Pugin (1762–1832) (after) John Bluck (fl. 1791–1819), Joseph Constantine Stadler (fl. 1780–1812), Thomas Sutherland (1785–1838), J. Hill, and Harraden (aquatint engravers).

But it is precisely because we are capable of automating more complex tasks than ever before, that now is a good time to introduce guidance in this area. In short, automated decision support systems are those that either recommend or decide on a course of action about an individual or business, often as they apply for a benefit or service from a federal institution. The decision almost always flows from authority provided in legislation or regulation. Renewing a passport, licensing a natural health product, rating a pilot for a certain aircraft, or granting a patent, are all examples of administrative decisions. Statistical research, for example, is not. Each have different risks to society associated with them, impact peoples’ lives and livelihoods differently, and require varying levels of “paper burden”/bureaucratic effort to process.

The system doesn’t necessarily need to make the decision, it needs to support it in a direct and meaningful fashion. So a human can still provide the final nod, but if a machine provides an assessment of risk, for example, then it can be covered. That said, this scope can quickly spiral out of control. Does an AI system that reads incoming mail and provide it to the right desk analyst based on its content constitute an “administrative decision support system?” After all, timeliness of service can impact how a decision was made. The answer is likely no; there needs to be some closeness between the system and the rendering of the decision, otherwise I risk capturing every system the government uses, even though this is not the spirit or objective of the exercise.

Why use these systems, given the risk? Because the reward seems to be worth it. I’ve written about this before, but I’ll restate a couple of reasons:

  • Humans decision making is problematic too. We make decisions on uninterpretable hunches, show unconscious biases, and fluctuate in our analytical capacity over a given day, much less our lifetime. Machines can reduce this inconsistency.
  • Machines can process applications much faster than people can. Is it ethical to make someone wait for 30 days to receive a notification of eligibility for a service, especially if we can give them an answer in a day?

So there is clear motivation to pursue this technology. But how to maximize benefits and minimize drawbacks?

Playing Pitfall!

I was seven when I first played the notorious Atari game Pitfall! at my cousin’s house in Toronto. Navigating this policy is like taking control of Pitfall Harry again and dreading the alligators and spike pits to come. Here’s just some of the complexity we will need to face over the next few months. I should also mention here that this isn’t the only thing I’m working on over the next few months, either.

  • Application — Which institutions will be covered? The more covered, the more legislative nuances we’ll need to consider, or exemptions to keep track of.
  • Scope — Like I mentioned above, how do we design language around a scope that institutions will find helpful, or will interpretation be an overly subjective exercise?
  • Requirements — Interpretability of models. Control of training data. Transparency of automated service. Security — physical and cyber. System audits. Contingency systems. Most importantly, can we scale requirements to the degree of impact that automating could have on society?
  • Consequences — What happens if the rules aren’t followed? What is a meaningful consequence, especially when legislation already provides some? What is the right balance between ensuring compliance but also allowing risk-taking?
  • Roles and Responsibilities — What degree of central oversight is necessary, desired, and/or practical? What is the role of CIOs in an AI age, versus the emerging role of chief data officers, or more technology-minded business owners?

Each of these elements will bring a difficult discussion around security vs. transparency. Honestly, I’m looking forward to them. The era of algorithmic government should be one defined by difficult discussions, particularly ones where there is a tension of values. Pitfall! was one of the most popular games for the Atari 2600 precisely because it was fun to avoid overcome the titular obstacles.

To start, I called a group of around 50 public servants representing a wide array of departments and agencies for a brainstorming session. There was palpable passion in the room around this subject, but overall the mix of data scientists, lawyers, ethicists, policy analysts and enterprise architects who participated generally supported the idea. I hope everyone does at the end of the day, but again, it is a long and winding road to the finish line. Of course, the devil is the details, and there will be so many in this Standard that you will think Goethe wrote it.

I personally believe that standard-setting, even in a limited context such as this, should be an open exercise. I endeavour to keep drafts of our work available to anyone interested in reviewing. This openness has led to a marked increase in quality of our AI white paper and I feel that it would lead to more sophisticated and nuanced policy. Will the Standard see the light of day? Who knows. Maybe we just won’t be able to come to agreement on an approach. Innovation doesn’t always succeed, but we’ll try and see what happens.

So to conclude a long post, I’ll leave a personal reflection. Rules don’t hinder innovation, they target it. They assist action. They fit in a “just do it!” type of world. But rules can only accomplish this if they are designed correctly. We have to take calculated risks in our rule-making, and be prepared to continually iterate and improve rather than drop text and let it go stale. Like the systems they seek to guide, these bodies of rules should be updated to fix bugs and include new features based on the feedback of their users. I sincerely hope that the end result is government AI that reflects the values of Canadians and works towards the best outcomes possible. We have powerful new tools at our disposal; let’s use them right.

Une version française de cet article sera disponible la semaine prochaine. Un lien sera affiché ici.