Evolve your design with optimal error rate — 15.87%
If you are not failing often, you are not trying hard enough.
During product design reviews, if you always receive positive feedback, you might be not probing enough or you are not asking the right questions, therefore you learn nothing new. According to Information Theory, if nothing surprises you, you only get data not information. That’s why every time I walk out of a UX review session with a bunch of “looking great” kind of comments, I will remind myself to stay alert and be aware of confirmation bias. However, if you get overwhelmingly rejecting voices and negative feedback, you might find it frustrating because you think you screw things up.
I am interested in learning how to get a balance between “not trying hard enough” and “nothing is working for me, I’d better try something else”.
Learning in Product Design
Consider scenario#1. Suppose you are a product manager or product designer. You have a bunch of assumptions to test in the backlog. In order to get continuous learning, you need to decide how many new features you want to get feedback on every week. Scenario #2, now that you have built a successful product, and you are still innovating. How radical should the updates be for each release?
Intuitively, I would test as less new stuff as possible to single out the results for causation analysis. And I would be conservative to only make slight incremental changes to avoid screwing up previous success and upset my customers.
But on the other hand, someone may argue making radical change is the source of innovation, creativity is not likely to come from conventions.
How hard is not too hard? How radical is not surprising? We might have a scientific answer now — 15.87%.
According to a paper discussing optimal learning difficulty level for machine learning algorithm, there is a magical number of error rate, with which to feed the algorithm in order for it to achieve optimal learning speed, that is 15.87%. And that seems also working for animals and humans.
“Eighty Five Percent Rule” for artificial neural networks used in AI and biologically plausible neural networks is also proven to be applicable describe human and animal learning.[1]
How AI evolve
I am not pretending I know much about machine learning, artificial intelligence. But I am curious about how amazing AI has been breaking records after records in the recent years in different industries. And it has been already applied in design industry[3]. As a designer, I am interested in knowing how AI will impact the future user experience and why they are capable of doing that. How possible is that AlphaZero beat human in chess without learning from humankind’s hundreds of years’ experience but only from learning from itself in 3 days, and became the best player in the world in 40 days [2] .
I am not going into the technical details, but in general, in machine learning, human need to provide dataset to train the algorithm against known answers. At each iteration, the algorithm will be told whether it did right or wrong. If it did it right, it will keep the strategy, if not, it will pivot or even turn around. Overtime, it learns to do it with less mistakes until reaching desired accuracy. Sounds familiar? That’s how designers get to the bottom of what users really want, that’s how entrepreneurs search for product market fit.
In machine learning, steadily increasing the difficulty of training has proven useful for teaching large scale neural networks in a variety of tasks. So the machine also face a question: how difficult should the training dataset be? The answer given by this paper is that
The optimal error rate for training is 15.87%
This means, if the algorithm is making mistake 15.87% of the time, it will reach maximum accuracy in the shortest time. If the algorithm makes too less mistakes, for example 5% error rate, which means the training is too easy, it will evolve slower. If the training is too hard, causing the algorithm to make too many mistakes, it will just confuse it and slow down the learning and may never be able to learn anything. For example, in the chart below, yellow represents highest accuracy and blue represents the lowest, with the same1000 trials, algorithm trained with 0.15 error rate reaches maximum accuracy rate whereas algorithm with 0.5 error rate doesn’t move the needle.
Below is just another chart to illustrate the optimal effect of 15.87%. If the same algorithm is trained for the 1000 times, but with different error rate respectively: 0.16, 0.06, 0.36. As you can see, the algorithm with o.16 error rate reach higher relative precision at the end of the trial.
OK, enough data and charts. So what does this to do with designing products? After all, this is all about learning.
What we can learn from AI
My first reaction to this “15.87%” rule is that: it is kind of making sense. Forget about the complex calculation behind this paper. If you think about it in common sense, if I learn something too hard it may be frustrating and discouraging, and never know if I am going to make it at all. If I learn something too easy, I might just still in my comfort zone. I may not grow fast enough to catch up with the pace of the world. On the other hand, achieving 85% correct rate is equally important because it would help confirming and reinforcing what has been learned before.
Now this magic number just gives us an opportunity to quantify not only how hard it is enough but also the sweet spot of hardness. According to the research, this does not only apply to machine learning, but also apply to broader scenarios, such as game design, movie making, language learning. The right level of difficulty makes a game challenging and addicting instead of boring. Carefully plot the twist of a movie creates novelty but avoid becoming totally controversy. Making sure you are exposed to 15% of new vocabulary every time lets you become familiar with your previous learning and also boost the speed of learning.
My takeaway
Adding to my design toolset
I don’t think we should strictly stick to any design guideline or design principle, because every case is different, and we should derive our design accordingly. However, design principles would provide us some theoretical bases when we are making design decisions. For example, golden ratio for photographer and graphic designer; 4 item limit working memory capacity for UI and interaction design, 5 user interview rules for UX research. Along with other unquantified design principle like Usability Heuristic, Gestalt Principles they make up of my design toolset. Because there are some many of them, the quantified principles are easier to remember and always bubble through the top of my mind. This 15% rule could potentially help me to decide what percentage of new features should roll out for one release. The next time when I design a user journey, planning 15% percentage of surprising events may be safe to create a sense of serendipity but not freak users out. Again, this is just adding it to one of the toolsets as hammer, I don’t have to see everything as nails.
Gauging learning pace
I knew that I need to try something hard and something new in order to advance. But I didn’t know in what pace will benefit me in the long run. I once experienced high desire to learn every aspect of design. I read all about design with full capacity everyday but soon discovering I wasn’t improving because a lot of content I consume start to overlap or repeat themself. My brains started to get used to similar topics and became insensitive. I was not getting new ideas, I felt like swimming in a well. Something changed when I started to learn about unrelated topics, my brain started to react to them and always circle back to how these ‘useless’ information could help me in my domain. However, overly spend time on unrelated stuff may become procrastinating. Now the 15% rule provides some guidance on how I allocate my attention and interest. Next time when I make some mistakes, upset some of my users, get some bad reviews, I would remind myself not to worry too much if it is within range, like 15%. Further more, if I am not seeing enough of them, I may make little progress.
Summary
Human brains certainly work fundamentally different than machines, humans will try to understand and explain what has happened, machines don’t. They just keep trying, until they get the desired results. However, this enables machines to evolve at a speed that humans could never achieve. Getting to know this magical number feels like we time travel to see the result of an experiment which we could not possibly finish in our life time.
What does 15.87% mean to you? Where could it potentially be applied to? Let me know what you plan to do with this number 🤘
Reference:
[1] Robert C. Wilsona,1, Amitai Shenhavb,c, Mark Stracciad , and Jonathan D. Cohene The Eighty Five Percent Rule for Optimal Learning https://www.biorxiv.org/content/10.1101/255182v1
[2]AlphaGo Zero: Learning from scratch https://deepmind.com/blog/alphago-zero-learning-scratch/
[3]https://www.alibabacloud.com/blog/alibaba-luban-ai-based-graphic-design-tool_594294
