A note to Data Science and Machine Learning managers
Use heuristics particularly those that direct you on what NOT to do
I am not a person that is big on advice. People are quick to dispense with this often worthless commodity, in fact it’s a rather large industry. Bookshelves are littered with the how-to self-help seven steps to this and that. Snake oil it is! In the world of money management, I found the way to cut through the clutter was to not to ask what should I be doing on the investment front, which almost always resulted in some normative advice, but rather what are YOU doing — show me where YOUR money is. Interestingly and unsurprisingly the advice often looked rather different to what the person giving it was actually doing. What they were doing told me all I needed to know. The advice was all too often worthless.
I will therefore dispense with advice in this note and rather focus on heuristics which I have found to be rather useful in complex environments particularly those that direct you on what NOT to do. In chess for example, grand masters focus on avoiding errors, rookies try to win. As pointed out by (Taleb, Goldstein, Spitznagel, 2009) recommendations of the “don’t” kind are usually more robust than “dos.”
So herewith is my little ensemble of the rules of thumb I apply of the “don’t” kind:
Don’t look for employees, favour entrepreneurial people
How can you possibly clearly delineate between an entrepreneurial person and an employee you ask? I have found it to be rather easy. Entrepreneurial people, after looking at all the reasons why something does not work, look to how they can make it work and then do it. Employees on the other hand look at all the reasons why something does not work and use it as a reason why to leave it at that. They are pretty easy to spot. A statement like; “My spouse and I both have great salaries that are difficult to walk away from for a maybe.” Is a sure sign that you are dealing with an employee. If your organisation is aiming to make an impact entrepreneurial people are going to best suited for doing such.
Don’t control, inspire
If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea. (de Saint-Exupery, 2003)
Try to figure out how to get great outcomes by setting the appropriate context, rather than by trying to control people. Entrepreneurial people thrive on freedom, they need resources to succeed and the organisational focus needs to be on empowering these people. This contrasts with employees who tend to consume resources and need to be managed. Challenges arise as the organisation grows and processes are used to combat complexity from scaling. This has a tendency to cripple innovation and drive out entrepreneurial people. To combat this it is critical to not lose focus on hiring the right people and to avoid adding processes and new rules. Using the Netflix vacation policy as an example; “there is no policy or tracking. There is also no clothing policy at Netflix, but no one comes to work naked. Lesson: you don’t need policies for everything.” (Netflix, 2009)
Don’t pretend you know the future
Business planning is the providence of charlatans. As Yogi Berra once said; “It’s tough to make predictions, especially about the future.” Business plans, business cases and things like belong on the dung heap. The real world is far too dynamic to conform to your procrustean bed no matter how flexible you think it is. The success of your organisation will depend more on the people in it and less on your glossy documentation.
Don’t be a sucker for the Planning Fallacy
In his book Thinking Fast and Slow, Daniel Kahneman provides some interesting examples in support of what he calls the “Planning Fallacy”:
- 1997 proposed new Scottish Parliament building 40 million (pounds). Completed in 2004 at roughly 431 million (pounds)
- 2005 study of rail projects undertaken worldwide between 1969 and 1998: 90% of cases the number of passengers projected to use the system was overestimated. On average planners overestimated how many people would use the new rail project by 106%. Average cost overrun was 45%
- 2002 American survey of homeowners found that the cost to remodel the kitchen: Average expected: $ 18 658, Average realised: $ 38 769
(Kahneman, 2013)
Moral of the story here is be sceptical of projected timeframes, budgets etc. and prepare yourself for things not going according to plan.
Don’t make survival contingent on a single outcome, make optionality your friend
I shall recount here the story of Thales and the olive presses. Thales was a philosopher, a Greek-speaking Ionian of Phoenician stock from the coastal town of Miletus in Asia Minor. We know about Thales, in part, because of the writings of Aristotle.
In Aristotle’s words the story is as follows:
“When they reproached him [Thales] because of his poverty, as though philosophy were no use, it is said that, having observed through his study of the heavenly bodies that there would be a large olive crop, he raised a little capital while it was still winter, and paid deposits on all the olive presses in Miletus and Chios, hiring them cheaply because no one bid against him. When the appropriate time came there was a sudden rush of requests for the presses; he then hired them out on his own terms and so made a large profit, thus demonstrating that it is easy for philosophers to be rich, if they wish, but that it is not in this that they are interested.” (Pessin, Morris Engel, 2015)
Aristotle’s reasoning for Thales success was in Thales superior knowledge “…having observed through his study of the heavenly bodies that there would be a large olive crop, …”
The reality is that Thales had entered into what appears to be the first option on record. You see Thales had the right, but not the obligation, to use the olive presses in case there would be a surge in demand. The providers of the olive presses had the obligation, not the right. Thales paid a small price for that privilege, with a limited loss and large possible outcome. (Taleb, 2014)
Don’t depend on intuition
I like to use the Monty Hall problem as an example here.
Let’s pretend we are on a game show. There are three doors and behind one of the doors is a prize, in this case a new car. There is a goat behind each of the other doors. The game show host knows what is behind each of the doors. The game show host asks you to pick a door and you choose door number 2. The probability you have selected the door with the car behind it is 1/3rd.
Now before revealing to you what is behind door number 2 the game show host opens one of the doors that you did not choose, door number 3, to reveal a goat.
There are two doors yet to be revealed, door number 1 and the door you chose, door number 2. Now knowing what you know, do you want to switch your choice? Does it matter? Will switching your choice improve your chances of winning? It appears that your chances of winning are 50% — two doors select one of them. But you would be wrong. Switching your choice to door number 1 at this point would double your chances of winning the car.
How is this? Since we started out with two goats it is more likely that what was behind your first choice, door number 2, was a goat — probability of selecting a goat at the start is 2/3rds. By revealing a goat behind one of the doors you did not select, the game show host is doubling your chances of winning if you switch your choice.
Don’t overestimate your ability to measure [1]
Not everything that can be counted counts, and not everything that counts can be counted. (O’Toole, 2010)
Don’t allow decision makers to make decisions without skin-in-the-game
In an effort to counter moral hazard in the presence of informational opacity the skin-in-game heuristic should be employed. Simply put anyone involved in an action which can possibly generate harm for others, even probabilistically, should be required to be exposed to some damage, regardless of context. (Taleb, Sandis, 2013)
Don’t choose between technical and business, you need both
In his books Secrets of Analytical Leaders, Wayne Eckerson speaks about what he calls purple people. Eckerson describes them as:
“…..people who live at the confluence of disparate approaches and opinions have a broader perspective. They see connections and possibilities that others miss. They speak multiple languages and gracefully move between different groups and norms. They continuously translate, synthesize, and unify. As a result, they imagine new ways to solve old problems, and they reinvent old ways to tackle new challenges. They are powerful change agents and value creators.
In the world of analytics, I call these men and women “purple people”. They are not “blue” in the business or “red” in technology, but a blend of the two, hence purple.” (Eckerson, 2012)
Purely technical people do not have the full understanding of the business problem to be addressed and purely business people do not have the technical expertise to derive a solution to the problem. It is these purple people that deliver the most value for the organisation.
Don’t grow to affectionate for your creations
Most people use statistics the way a drunkard uses a lamp post, more for support than illumination. (O’Toole, 2014)
Although primarily directed at those in the quantitative finance community, the financial modellers manifesto can be extended beyond and is relevant to those in the field of modelling in general:
The Modelers’ Hippocratic Oath (Derman, Wilmott, 2009)
- I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
- Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
- I will never sacrifice reality for elegance without explaining why I have done so.
- Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
- I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.
Appendix:
[1]
Frank Knight may also be of consideration here:
The saying often quoted from Lord Kelvin (though the substance, I believe, is much older) that “where you cannot measure your knowledge is meagre and un-satisfactory,” as applied in mental and social science, is misleading and pernicious. This is another way of saying that these sciences are not sciences in the sense of physical science, and cannot attempt to be such, without forfeiting their proper nature and function. Insistence on a concretely quantitative economics means the use of statistics of physical magnitudes, whose economic meaning and significance is uncertain and dubious. (Even “wheat” is approximately homogeneous only if measured in economic terms.) And a similar statement would apply even more to other social sciences. In this field, the Kelvin dictum very largely means in practice, “if you cannot measure, measure anyhow!” That is, one either performs some other operation and calls it measurement or measures something else instead of what is ostensibly under discussion, and usually not a social phenomena. To call averaging estimates, or guesses, measurement seems to be merely embezzling a word for its prestige value. And it might be pointed out also that in the field of human interests and relationships much of our most important knowledge is inherently nonquantitative, and could not conceivably be put in quantitative form without being destroyed. Perhaps we do not “know” that our friends really are our friends; in any case an attempt to measure their friendship would hardly make the knowledge either more certain or more “satisfactory”!
Reference:
de Saint-Exupery, A. (2003). The wisdom of sands. [book]. Amereon Ltd. [ISBN-13: 978–0848825959]
Derman, E. Wilmott, P. (2009). The financial modeler’s manifesto. [pdf]. Retrieved from http://www.uio.no/studier/emner/sv/oekonomi/ECON4135/h09/undervisningsmateriale/FinancialModelersManifesto.pdf
Eckerson, W. (2012). Secrets of analytical leaders: insights from information insiders (1st ed.). Technics Publications, LLC. [ISBN 10: 1935504347]
Kahneman, D. (2013). Thinking fast and slow. [book]. Farrar, Straus and Giroux. [ISBN-13: 978–0374533557]
Knight, F. (1940). “What is truth” in economics?. [pdf]. The University of Chicago Press. Retrieved from http://www.jstor.org/stable/1825908
Netflix. (2009). Freedom and responsibility. [pdf]. Retrieved from http://www.slideshare.net/reed2001/culture-1798664
O’Toole, G. (2010). Not everything that counts can be counted. [website]. Retrieved from http://quoteinvestigator.com/2010/05/26/everything-counts-einstein/
O’Toole, G. (2014). Most people use statistics the way a drunkard uses a lamp post, more for support than illumination. [website]. Retrieved from http://quoteinvestigator.com/2014/01/15/stats-drunk/
Pessin, A., Morris Engel, S. (2015). The study of philosophy: a text with readings. [book]. Rowman & Littlefield Publishers. [ISBN-13: 978–1442242821]
Taleb, N. (2014). Antifragile: things that gain from disorder (incerto). [book]. Random House. [ISBN-13: 978–0812979688]
Taleb, N., Sandis, C. (2013). The skin in the game heuristic for protection against tail events. [pdf]]. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2298292
Taleb, N., Goldstein. D., Spitznagel, M. (2009). The six mistakes executive make in risk management. [pdf]. Retrieved from https://hbr.org/2009/10/the-six-mistakes-executives-make-in-risk-management
