Reproducibility
The Theory
Reproducibility in and of itself refers to the ability of others to reproduce something that another person has already done. The traditional and probably most popular use of this word relates to scientific research or statistical analysis. This paper, however, will refer to its role in everyday life, particularly its role in allowing excess competition to freely flow to an occupation, industry, or skillset. To better define and explain this, we will have to look to some obvious and some less obvious examples.
Certain things in the world appear to be more reproducible than others. Additionally, there is a difference between something that is not reproducible and something that is difficult to reproduce. There don’t seem to be many skills or knowledge one can acquire in the world that cannot be reproduced. Of those that do exists, most that I can think of are limited to the category of natural talent. Of course, not everyone can be LeBron James or Warren Buffett because they were born different than most. This does not discount the countless hours of practice and learning each has put into his craft (probably more than almost anybody else in their field), but it is a good example of results or skills that are probably not reproducible. Even if someone does everything that LeBron James or Warren Buffett does or tells them to do, they will almost certainly not be able to produce the same results as those two.
It is possible for someone to accomplish something that cannot be reproduced, but for most people, this is not an aspiration that is within their reach. In my opinion, reproducibility in and of itself is not an inherently bad thing or something to be avoided; it is almost unavoidable. What should be avoided is the acquisition of a skill or knowledge that can be reproduced faster and cheaper by others (I’ll call this short-form reproducibility). For instance, say someone spends countless hours, years even, honing their skill as a fundamental equity analyst and stock selector. This skill can be valuable to many, but what happens when someone else figures out how to reproduce this method of stock selection through quantitative methods that are much cheaper and faster? All those years of accumulating specialized knowledge and skills can be bypassed by a newcomer with quantitative investing know-how (i.e. there are low to no barriers to entry). Obviously, many quantitative processes cannot successfully capture all of nuances that a human analyst can, but in terms of taking clients and delivering alpha, people have found cheaper and faster ways of reproducing the results of the human learning.
This can also be seen in the context of a university education. Depending on the field of study one pursues, it is arguable that learning only in the context of a classroom or collegiate environment does not give one any kind of competitive advantage. Yes, someone may be top of their class, but there are many more coming behind them who will, by default, learn the same things in the same classes, possibly in new and more efficient ways. Now, this may not be an example of continuously faster acquisition of a skill set, but it is an example of a form of commoditization or “table stakes.” The skill sets that are learned in this setting are reproducible because they are acquired in an environment that encourages reproducibility.
A reasonable approach to this dilemma for someone who does not possess a genius-level IQ or genetic superiority is to acquire a skill or knowledge that can only be reproduced by the same painstakingly tedious and time-consuming methods by which they acquired it (I’ll call this long-form reproducibility). This means to look for skills or knowledge that can be learned by doing something most other people aren’t doing and can only be learned by someone else through taking the same steps that you took to achieve a form of fluency. This is easier said than done, and many skills or forms of knowledge that appear to be only long-form reproducible end up being short-form reproducible in the future.
The most obvious and common way to apply long-form reproducibility is to acquire specialized knowledge that is learned through unique experience and/or long periods of study and can only be learned by someone else through this same method. This may sound exactly like the university environment method of learning but is different in that this refers to a non-standardized and personal way to obtain knowledge or acquire a skill. This form of learning can refer to learning in a way that doesn’t involve paying someone else to give you the knowledge or teach you the skills. While this may (in theory) limit the number of people who can acquire this skillset or knowledge, it also commoditizes it and incentivizes the teacher of it to teach it to as many people as possible. Of course, all forms of learning are paid for in some way or another, but a long-form reproducible method usually involves learning outside of a setting that encourages others to reproduce the method and in which the way one learns it is the only way to learn it.
This method also does not only have to apply to niche areas. Charlie Munger’s latticework of mental model method of analyzing and learning about the world is a more well-known application of a long-form reproducible learning method. Someone cannot attain and build an internal repertoire of knowledge spanning different disciplines without taking the time to learn about and understand those disparate disciplines and how they fit together and build off one another. This is not something that is standardized and taught in schools. Unless someone is especially adept at high-speed learning, they cannot attain this level and breadth of knowledge during a four or five-year span at a university. It is something that will take much longer and can only be accomplished in one way (that I know of).
The same can be said of Patrick O’Shaughnessy’s extensive list of books he is able to read every year. Those books contain reams of knowledge that are only attainable through the same form of consumption (again, that I know of). Someone cannot acquire this type of knowledge without going down the same path he did. There is no way to speed up the process or complete it in a more efficient manner, you simply must read all the books.
Possible Pitfalls
This type of learning, however, should only be applied in the right settings. There are times when knowledge or skills that are acquired only through experience or some other long-form reproducible method can be harmful to apply. There is a difference between an understanding and internal indexation of knowledge or skillsets and their application in decision making. This usually occurs when the sample size of experience or depth of knowledge or skills attained is too small to be applied in a consistent decision-making environment. This does not mean that one must attain a greater breadth of knowledge or skills to be able to make useful and accurate decisions; it means that the universe of knowledge or possible experience is so large that relying on the experience and accumulated knowledge of one person, or even several, will result in inadequate outcomes.
To illustrate, one can look to the practice of law, specifically litigation. A skilled and experienced litigator may be able to better navigate the different intricacies of the judicial system and formulate better arguments, methods of persuasion, or litigation strategies as he or she gains more experience and knowledge in the field. However, when it comes to making a decision to accept a case or pursue a claim, one cannot rely solely on their own experience in determining whether it will be successful. Take, for instance, the number of civil cases filed in a federal district court. Using the Southern District of Mississippi, as an example, in the twelve months from March 31st2017 to 2018, there were 1,736 civil cases commenced1. There were additionally 1,602 already pending as of March 31, 2017. This caseload does not appear to be particularly large compared to jurisdictions like the Eastern District of Louisiana where 18,066 cases were commenced over this same period and 23,163 were already pending as of March 31, 2017. Obviously, these cases are not separated into specific practice areas, but these statistics do not take into account the magnitude of state caseloads during this period (which is much greater than the numbers above). This also does not take into account settlements that are negotiated before filing and alternative dispute resolution (a very large number). The bottom line is that experience relied on by a sole litigator or even a firm of several does not appear to involve enough information to make an accurate decision based on this experience or accumulated knowledge.
Even a litigator with many years of experience will not have experienced enough cases to have a statistically significant amount of litigation outcomes in his or her head. The amount of cases the average litigator handles throughout his or her entire career will only be a small fraction of the cases litigated in their jurisdiction each year. This does not take into account the amount of cases litigated in other jurisdictions that may be applicable (even though the law may not be accepted as precedent) in deciding on whether to accept or not accept a case. In fact, in situations like this, the acquired long-form reproducible knowledge and experience may be harmful. This is not to say that the experience and acquired knowledge is not useful in other ways such as litigation strategy or argumentative enhancement; a sole reliance on it just may be harmful in a decision-making context. The flip side is that this experience and knowledge may be short-form reproducible in the decision-making context by the use of statistical methods or machine learning. In trying to avoid a descent into a poor attempt at an explanation that may be riddled with buzzwords, I will settle with stating that it seems apparent that some form of statistical application will be able to do (much better too) what practitioners do without going through the same steps of knowledge accumulation and skill development. Again, this is purely in reference to a decision-making environment, and relates, in much the same way, to the quantification of security selection.
Sticking with the legal profession, the same can be said of contract drafting in specific situations. While there may not be access to data in this situation, it could make a world of difference in decision making. When deciding how to draft a provision or to change a decade old template that your law firm has been using, it would be helpful to know how certain terms have fared. Unfortunately, many disputes arising out of disagreements over contracts are handled through arbitration or are confidential in some way, but firms could and might already make decisions regarding contract structure and language in light of data that has been collected in the course of practice. Considering the majority of civil cases litigated in the United States are contract disputes, making data-based decisions would save a substantial amount of money, wasted time, and headaches.
To conclude, I would just like to say this is my opinion and an observation that may or may not prove to be true. Obviously, nothing is as great a teacher as experience. One can read about some historical event in a book, such as a stock market crash, and think they have learned how to respond, but nothing will test their muster like experiencing one in real life. This illustrates how experience is often the only true teacher, but there also is a flip side. The experiences one has, especially at the beginning of one’s career, often shape how one views the world from then on. Take, for instance, a discretionary value investor starting out in the mid-to-late 90’s to early 2000s era. They would, and have, been struck with the conviction that value always outperforms in the end and high CAPE ratios are a warning flag. But who decides how long it will take for value to outperform and who decides what level of CAPE is too high? If you were a value manager in the early 2000’s, you would come to the conclusion that it usually takes about 3–5 years at the most for the market to come to the same realization that you have about the value of an undervalued company. But may I suggest that this may have just been related to the time period it took the market to recover from the TMT bubble crash and subsequent recession. This does not necessarily prove the same will be true over subsequent periods, and one could have observed this play out after 2008. Many called for a recession or crash just a few years after the recovery, and many underperformed while sticking to the same strategy that allowed them to outperform during the early-to-mid 2000’s period.
No matter how hard someone tries to or believes that they try to rely purely on hard data to make decisions, experiences shape how we think. This is a possible downside to the long-form reproducibility hypothesis and its usefulness — it is somewhat static. Learning and developing some form of possible expertise in an area that is only long-form reproducible has appeared to offer great returns on investment in the past, but in a constantly changing world it is possible that it could be a handicap in some situations.
1 Statistical tables documenting federal court cases filed in each district can be downloaded at https://www.uscourts.gov/statistics-reports/caseload-statistics-data-tables
