Will big data replace physicians?
Self-diagnosing an illness is becoming increasingly popular. Online there are sites that make this very easy for you. The quality however tends to be very low, as so much information is provided. In contrast to these websites, a physician is able to ask questions and to do additional tests. But could you state that providing the right information to smart algorithms would be better, in order to eliminate human error from the equation?
Decision tree diagnosis
What a physician does in general is following a decision tree based on questions for each of the presented symptoms, until enough confidence is acquired to make a diagnosis. How the branches and leafs are weighted and evaluated is dependent on knowledge, skill and experience of the physician. The society to improve diagnosis in medicine (SIDM) claims that 5–15% of all diagnosis are wrong (USA), indicating that some of these physician’s decision trees were trained poorly.
Like normal decision trees it is quite easy to overfit the train data, which increases the test error, or in this case increases the risk of misdiagnosing a patient. Why not combining these diagnosis steps (trees) of multiple physicians into a bigger general system (forest) thus improving the accuracy?
Diagnostic tests and Imaging
Diagnostic tests like drawing blood and analyzing it, is a process that can be carried out by smart algorithms. Even medical imaging like X-ray or MRI, for which some institutions already have smart machine learning to read and interpret them. As more of these techniques evolve and more data is available, machine learning techniques can gain far more experience than any physician could achieve in multiple lifetimes.
Data collection and storage
It will be a challenge to collect features that can capture behavior of deceases. Most behavior features that increase decease understanding invade personal lives extensively because you need to consistently monitor all patients even when they are in seemingly full health. Checkups only when people are sick would need to move to a more continuous data collection about personal health, like the use of medical-device wearables.
Data generated by the continuously monitoring needs to be stored somewhere or at least transformed into forms that can be used to analyze. The need of storing data of many people will only be possible with a big data infrastructure that can guarantee privacy.
Accepted machine learning take over
Information-power companies like Google and Netflix apply machine learning to predict what search results are most relevant to you or what kind of movie you like to watch. Their algorithms work well as a vast amount of people use their services. Clearly it doesn’t bother people that their movie-watch-behavior is used to predict what movie they want to watch next, probably because it helps them a great deal in finding the right movies to watch. McKinsey states these accepted ways of using big data can also be applied in medicine.
It might not be long before big data techniques could take over the work of physicians. Techniques we are already using, such as recommender systems, could probably be adapted to predict optimal treatments for deceases, though in my opinion it could still take a while before these data-driven decisions will be accepted. What do you think?
BigData Republic provides Big Data Solutions. We are experienced in advanced predictive modeling and deploying large-scale big data pipelines that take security and privacy into account. Interested in what we can do for you? Feel free to contact us.
