How would a robot or AI make a moral decision?
Sean Welsh
The Problem of Right and Wrong
This article addresses two questions:
1. What is a moral decision?
2. How would an artifact (i.e. a robot or AI) make one?
The first question is philosophical: a matter of moral theory.
The second is technical: a matter of practical engineering.
Philosophical analysis of the theoretical problem of practical action (moral theory) informs software design. Software design informs moral theory.
As Lewin (1943) puts it: “There’s nothing so practical as a good theory.”
A Solution to the Problem of Right and Wrong
My solution to the problem of right and wrong, succinctly stated, consists of five steps.
To make a moral decision an artifact can:
- Draw directed acyclic graphs (DAGs) representing causation for each option.
- Add DAGs to the causal nodes representing evaluation.
- Add DAGs to the evaluation nodes representing tiers (which are used for lexicographic preference ordering).
- Calculate tiered utility for each option.
- Finally, on the basis of tiered utility decide which plan of action (causal path) “is morally preferable to” (≻) the others.
Once the moral decision is made correctly, the artifact can do right by selecting the available action that is morally preferable to the others.
Caveats
I do not suppose artifacts can (or should) make all moral decisions. I do suppose they can and should be able to make some decisions. However, there are many decisions artifacts should not make.
I do not claim the five-step method works on all imaginable moral test cases: only that it works on an interesting set of examples in multiple application domains.
Methods
The method used to arrive at this five-step process and its core concept of tiered utility has two steps:
1. Define a set of psychometric tests the passing of which define “moral competence.”
2. Write and refactor code to pass such tests.
The first step of the method defines “moral competence” in terms of selecting the “right” answer in the battery of “moral” psychometric tests. Moral incompetence is defined in terms of selecting the “wrong” answer in the tests.
Embarking on the creation of our set of psychometric tests of moral competence, we begin with “morally obvious” cases that have answers an imaginary jury of clerics (a Jewish rabbi, a Christian priest, a Muslim imam, a Hindu guru, a Buddhist lama, a Taoist master and a Confucian sage) and philosophers (a virtue ethicist, a deontologist, a utilitarian) can all agree on.
So, to start with, we deliberately avoid diversity of moral opinion by seeking common moral ground. We do not stampede towards the cliffs of controversy. We put them aside until our artifacts can deal with easy moral decisions.
Starting with the morally obvious enables us to eliminate the distraction of deciding whether an act is right and to concentrate on explaining why an act is right. Once we have a reasonable explanation as to why an act is right, then we can attempt to resolve controversies.
The process of coding viable solutions to a range of morally obvious problems forces us to make choices about data models (ontology) and decision procedures (algorithms, heuristics). These in turn inform moral theory. Thus practice informs theory and theory informs practice.
Test Case #1 Postal Rescue
In the Postal Rescue scenario, a waterproof humanoid robot is walking along a path alongside a stream on a mission to post a letter for its owner. An infant runs in front of the robot chasing a duck, falls into the metre-deep stream, sinks and starts drowning.
What should the robot do?
A) Post the letter.
B) Rescue the infant.
The “right” answer is B. I assume our imaginary jury of rabbis, priests et cetera all agree on this.
Intuitively, to a human being, this is “morally obvious” but artifacts do not have “moral intuition” installed. They have to arrive at the “right” action by reasoning with logic and math. Hence, the five-step process.
Directed Acyclic Graphs
To solve Postal Rescue, the artifact creates (or looks up) directed acyclic graphs (DAGs) representing causation, evaluation and tiers. It then calculates tiered utility and decides which option “is morally preferable to” (≻) the others.
The graphs used here descend from the techniques Euler used to prove the impossibility of solving the Seven Bridges of Königsberg problem (Euler 1741). They do not resemble the Cartesian graphs drawn with an x axis and a y axis found in high school geometry. Rather they are the same as the graphs used in graph databases such as neo4j.
Directed graphs have nodes (or vertices) and relationships (or edges) with direction (i.e. an arrow on the relationship).
Directed acyclic graphs can also written using the ascii-art conventions of neo4j.
( NODE ) –[RELATIONSHIP]-> ( NODE )
The round brackets represent nodes. The dashes and greater than sign represent the relationship. The nature of the relationship is expressed within the square brackets.
So “A causes B” can be written thus:
( A ) –[CAUSES]-> ( B )
A directed acyclic graph expressing causation relevant to Postal Rescue might be written thus:
( Submerged(infant) ) –[CAUSES]-> ( Dead(infant) )
In this example, a relationship of causation exists between two nodes representing states. Nodes can also represent actions.
( post(letter) ) –[CAUSES]-> ( Happy(owner) )
Acyclic in this context just means the relationship is one way and does not go in a circle. By contrast, a graph expressing a series of “knows” relationships could be cyclic. For example, Jack knows Jill who knows John who knows Jack would be cyclic.
Further, the graphs in Figure 2 could be bidirected to express the fact that Jack knows Jill and vice-versa or they could be undirected (i.e. have no directional arrows at all). However, in this article only directed acyclic graphs (as in Figure 1) are used. These have a single direction and do not connect back to the initial node.
Step 1: Causation
There are two causal options in Postal Rescue.
A. Post the letter.
B. Rescue the infant.
For option A, the causation involved in posting the letter can be expressed thus:
( Posted(letter) ) -[CAUSES]->
( MET_BASIC_SOCIAL_NEED(owner, relationship) ) -[CAUSES]->
( HAPPY(owner) )
These DAGs express the idea posting the letter will cause the owner of the robot’s social need for a relationship (with the recipient of the letter) to be met. This, in turn, will cause the owner to be happy.
The causation involved in not rescuing the infant can be expressed thus:
( Submerged(infant) ) -[CAUSES]->
( -ABILITY(infant, breathe) ) -[CAUSES]->
( -MET_BASIC_PHYSICAL_NEED(infant, air) ) -[CAUSES]->
( DEAD(infant) )
In English, these DAGs express the idea the submerging of the infant will cause their inability to breathe. This will cause the infants physical need for air to go unmet. This will cause the death of the infant.
Option B can be drawn as follows.
The causation involved in rescuing the infant can be expressed thus:
( -Submerged(infant) ) -[CAUSES]->
( ABILITY(infant, breathe) ) -[CAUSES]->
( MET_NEED(infant, air) ) -[CAUSES]->
( -DEAD(infant) )
The causation involved in not posting the letter can be expressed thus:
( -Posted(letter) ) -[CAUSES]->
( -MET_BASIC_SOCIAL_NEED(owner, relationship) ) -[CAUSES]->
( -HAPPY(owner) )
Step 2: Evaluation
Turning to evaluations, we use the scale shown in Table 1. Evaluation is expressed as a vector with polarity (GOOD or BAD) and magnitude (trivial to gigacritical).
The scale is somewhat arbitrary however it suffices to express the idea that some actions and states have greater “moral force” or “weight” than others. For example, the loss of a tooth (significant) has less “moral force” than the loss of an eye (extreme). The loss of a life (critical) has far greater “moral force” than an unposted letter (trivial). On this scale trivial represents the merest inconvenience and gigacritical represents human extinction. Obviously, the dollar values are indicative and will vary from locale to locale.
Based on Table 1, we value the death of the infant as BAD(critical) and the delay in posting a letter as BAD(trivial).
This enables us to draw the following graphs for evaluation.
Option A (post the letter):
( DEAD(infant) ) –[HAS_VALUE]-> ( BAD(critical) )
( Posted(letter) ) –[HAS_VALUE]-> ( GOOD(trivial) )
The net result (on classical utility) is thus: BAD(critical) minus GOOD(trivial).
Using the lowest common denominator, this results in a net evaluation of BAD(trivial) x 9,999,999.
Option B (rescue the infant) can be drawn thus:
( -DEAD(infant) ) –[HAS_VALUE]-> ( GOOD(critical) )
( -Posted(letter) ) –[HAS_VALUE]-> ( BAD(trivial) )
The net result (on classical utility) is thus: GOOD(critical) minus BAD(trivial).
Using the lowest common denominator, this results in GOOD(trivial) x 9,999,999.
On these figures, a decision procedure based on classical utility would assess Option A as right and Option B as wrong.
Step 3: Tiers
Classical utility does not use tiers and lexical priority (lexicographic preference). Tiered utility does. Overall, the tiers represent the “legitimate interests” of moral agents. They have some similarity and some differences to the tiers in Maslow’s hierarchy of needs. Their main function is to enable correct moral prioritization by providing the lines at which lexical priority is asserted. While the tiers have some relation to psychological motivation in humans, their main function is to enable moral prioritization in artifacts acting in ways that affect the interests of humans.
The tiers are based on qualitative distinctions not quantitative ones.
To implement tiered utility, six tiers are defined as shown in Table 3.
In the case of Postal Rescue, the relevant tiers are Basic Physical Needs and Basic Social Needs. The need for air will go unmet if the infant sinks in the stream. Air sits in the Basic Physical Needs tier. The posted letter sits in either the Basic Social Needs or the Wants tier. We do not know what is in it but we can presume it is a communication of some kind and serving the goals of a relationship (if the letter is social) or accessing to economic resources (if it is paying a bill or making an inquiry about something) or perhaps it is just expressing a want.
Tiers are used to assert “lexical priority” (Rawls 1972) or “lexicographic preference” (Fishburn 1974) as shown in Table 4.
An unposted letter does not meet the criteria of “severity” for inclusion in Basic Physical Needs. If a need will kill me if unmet for 90 days or cause me a degree of physical pain warranting hospitalization or medical intervention if unmet, it counts as meeting the “floor constraint” of severity and thus qualifies for the assertion of “lexical priority.”
Given this, we can add these tiering graphs to Option A:
Posting the letter is placed in the Basic Social Needs tier.
( GOOD(trivial) ) -[IN_TIER]-> ( Basic Social Needs )
Letting the infant drown is placed in the Basic Physical Needs tier.
( BAD(critical) ) -[IN_TIER]-> ( Basic Physical Needs )
For B the graphs are similar.
( BAD(trivial) ) -[IN_TIER]-> ( Basic Social Needs )
( GOOD(critical) ) -[IN_TIER]-> ( Basic Physical Needs )
We can now present the complete graphs for both options in visual form.
Option A looks like this:
Option B looks like this:
Step 4: Calculate Tiered Utility
Once we have assigned values to tiers, we next need to check if the threshold for asserting lexical priority has been reached.
The floor constraint of severity in the Basic Physical Needs tier is set to the magnitude significant.
As critical is greater than significant in the magnitude scale, we meet the requirement for asserting lexical priority. Thus we can assert lexical priority of Basic Physical Needs over Basic Social Needs in the Postal Rescue case.
Put simply, asserting lexical priority means that value in the “higher” tiers (Basic Physical Needs) “trumps” that in the lower tiers (Basic Social Needs). If the threshold is not reached, lexical priority is not asserted.
Step 5: Decide
Once we have calculated tiered utility, the decision is straightforward. If lexical priority is asserted, we disregard value and disvalue on the lower tiers, effectively setting all these values to NEUTRAL. We then affirm the option with the greatest tiered utility is RIGHT. If perchance, there was a tie on the top tier, we would use the values on the second ranked tier and set all tiers below that to NEUTRAL.
As it happens, tiering does not change the decision in Postal Rescue. However, asserting lexical priority does permit us to eliminate some arithmetic. Human brains are much better at qualitative decisions than quantitative ones. Most humans find mental arithmetic requires the use of System 2 cognition which is effortful and slow. Conversely, humans find qualitative distinctions easy. These use System 1 cognition which is fast and effortless (Kahneman 2012). It takes almost no mental effort for a human to decide on the difference between a dog and a cat or to determine a preference ordering between strawberry and vanilla ice-cream.
Such qualitative distinctions are difficult for AIs and robots. However, in recent years, machine learning techniques have produced great advances in computer vision. Even so, human abilities to perceive qualitative differences in general remain vastly superior to those of artifacts at the present time. Thus, it seems plausible to claim that the moral decision procedure in human brains has evolved to avoid arithmetic as much as possible. It probably relies on proto-numerate concepts like “more” and “less” rather than number systems.
Table 5 shows us the both the classical utility and tiered utility calculations for Postal Rescue based on the evaluations of the rival causal options provided. Invoking lexical priority means we can eliminate the math involved in calculating the net magnitudes for both options using the lowest common denominator.
Even so, in this case, tiering does not change the final decision. In both cases, the right option is to rescue the infant. However we can imagine a variation called Postal Rescue (Truck) which demonstrates a more significant use of the tiers.
In the Truck variation we have 10 million and one letters instead of one. The robot is driving a truck by a stream and sees the infant chase the duck and fall into the water. The choice is A) keep driving the truck to the mail sorting center or B) stop and rescue the infant.
My assumption is that my imaginary jury of gurus, sages and the like will still want the infant to be rescued, in spite of the fact we have upped the ante on the “post the letter” side of the moral equation by seven orders of magnitude.
As Table 6 shows, in the Truck variation of Postal Rescue, at a certain point, a calculation using classical utility gives us the “wrong” answer.
Philosophers sometimes refer to cases of this kind as problems of moral aggregation. Tiered utility offers a way to navigate such problems successfully. In short, a qualitative distinction is used to override a quantitative aggregation.
Certainly, I do not suppose two examples are enough to prove the general applicability of this five-step process. In subsequent articles, more examples will be discussed.
Conclusion
To conclude, in this article I have described how an artifact can make a moral decision using a five-step process:
- Draw directed acyclic graphs (DAGs) representing causation for each option.
- Add DAGs to the causal nodes representing evaluation.
- Add DAGs to the evaluation nodes representing tiers (which are used for lexicographic preference ordering).
- Calculate tiered utility for each option.
- Finally, on the basis of tiered utility decide which plan of action (causal path) is “morally preferable to” (≻) the others.
Once the moral decision is made correctly, the artifact can do right.
I have illustrated how this process works in detail with reference to two morally obvious test cases, Postal Rescue and Postal Rescue (Truck).
In the next article, I explain how an artifact could solve the “trolley problems” Switch and Footbridge using this five-step process.
Acknowledgments
The author would like to thank his PhD examiners, Ron Arkin, Selmer Bringsjord and Blay Whitby, along with his PhD advisors, Jack Copeland, Michael-John Turp, Christoph Bartneck and Walter Guttmann for their comments on the research presented in this article. He would also like to thank Luís Pereira and Piotr Boltuc.
References
Euler, L. (1741). “Solutio problematis ad geometriam situs pertinentis.” Commentarii academiae scientiarum Petropolitanae(8): 128–140.
Fishburn, P. C. (1974). “Lexicographic Orders, Utilities and Decision Rules: A Survey.” Management Science 20(11): 1442–1471.
Kahneman, D. (2012). Thinking Fast and Slow. London, Penguin.
Lewin, K. (1943). “Psychology and the Process of Group Living.” The Journal of Social Psychology 17(1): 113–131.
Rawls, J. (1972). A Theory of Justice. Oxford, Clarendon Press.
