Why do Many find Machine Learning less Satisfying than Standard Coding?

A question originally posted on Quora; answer x-posted here to help educate those still uncertain how machine learning changes how we build software.
Machine learning isn’t less satisfying for everyone. In fact for some it is much more satisfying, and the camp you fall into depends on how you like to approach solving problems.
The world of software development now has 2 choices when it comes to producing software behavior:
Inside Out programming: manually write explicit rules to produce the behavior with which users interact;
Outside In programming: train computers to consistently produce the behavior they see in data.
The first option comes from traditional/standard computer programming. Here, the programmer is in full control over how the system produces its output. This is how engineering has been defined since the industrial revolution. We think deep about each part of the system and how they interact. We write all the rules of the game, test that those rules don’t deviate from expected behavior, and try to manage the growing complexity of our systems using additional rules and design patterns.
The second option comes from machine learning. Here we cede control of the rule-writing to machines. We do this because many of the challenges we now face in software cannot be solved by writing rules humans understand. Try using standard programming to recognize a face in an image, the identity of a person speaking, a malicious piece of code, an optimal financial portfolio, to create a self-driving car, surface the best diagnosis for a patient, the best chemical candidates for a new drug, the topics that exist across thousands of documents, structure a profitable marketing campaign, forecast workplace accidents, or create a chat service that can repsond to a seemingly infinite number and type of questions.
We cannot approach today’s biggest challenges by writing explicit rules using standard coding. It is beyond our capacity to discover what those rules are and therefore we need to accept that our reductionist style of problem solving is rapidly becoming a dated approach to engineering systems.
Accepting the “black box” nature of machine learning does not come naturally to those who have been taught that engineering is about knowing how systems work at their deepest level, and that by hard-coding rules we can produce machines that solve problems. But it does come naturally to those who are used to using technology at a higher level of abstraction, folding in things like human behavior and other softer concepts into the process of making software products. These individuals don’t try to reveal the fundamental basic laws, rather they are more strategic and conceptual in their approach to solving problems. So your comfort level can come from how you think about using technology.
This can come as a shock to those looking to get into machine learning, and particulalry Data Science where we solve very strategic challenges using software that explains and predicts using its own learned rules. The skillsets needed in a field like Data Science have to do with learning how to work with machines, rather than how to build them from the ground up.
So machine learning is more satisfying for many because you can solve much more complex problems than with traditional rules-based programming. It is also less satisfying for those who want explicit control over how each piece of programming produces its results. Of course software will always need both approaches, but standard coding is being relegated to the scaffolding that holds the machine learning in place.
The 2 modern roles for building data-driven software are Data Engineering and Data Science. My advice for those looking to get into data is to decide which camp, as per above, you are more comfortable in. Data Engineering is closer to the standard approach since this area builds systems to keep machine learning operating smoothly, and to get it working with the other more traditional pieces of the application. Data Science is more about the art of getting machines to write great rules, based on data we have gathered and prepared, and our ability to validate models both statistically and based on user feedback. Data Science blends mathematical understanding with behavioral concepts, creating software that adapts to its surroundings via the data it is exposed to; not using explicit rules coded by people.
Of course your role will always blend a bit of both camps but it is important to decide how you like to approach problems. Machine learning is a fundamentally different way to create software, and we are only beginning to understand how to bring its potential into the traditional world of rules-based programming. There is a lot of friction at the intersection of these 2 camps, but with friction comes an awesome opportunity to make better products.
