ChatGPT’s Near-Passing Score on US Medical Exam: What Are the Potential Downfalls?

According to a recent study, ChatGPT achieved near-passing scores on the US medical licensing exam(USLME), indicating potential applications for AI in the healthcare industry.

The USLME is renowned for its challenging nature, which typically necessitates 300 to 400 hours of preparation and encompasses a wide range of subjects ranging from basic scientific concepts to bioethics.

The exam consists of three tests, and ChatGPT’s proficiency in answering its questions indicates that AI bots could become valuable in medical education and even in making certain types of diagnoses one day.

“ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement,” write the researchers in their published paper. “Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations.”

During the assessment, ChatGPT achieved a score of 52.4% to 75% in the three exams (the passing grade is about 60%). In 88.9% of its responses, ChatGPT generated significant insights that were termed “new, non-obvious, and clinically valid” by the researchers.

“Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation,” the study authors said in a press statement.

While the ability of machine learning tools to answer medical questions as accurately as humans are promising, such as providing help to improve the quality of care, increase efficiency, and reduce costs, experts in the field highlight the limitations and ethical considerations that must be addressed.

One of the most pressing concerns is the potential for biases in machine learning algorithms. which can lead to discrimination against certain groups of patients. Biases can arise during the algorithm development process or may be introduced due to underlying data sets that are not representative of the population. It is crucial to ensure that machine learning algorithms are thoroughly tested for bias and that diverse data sets are used during development. Failure to do so could result in significant harm to patients and perpetuate existing health disparities.

Another is the potential for machines to misinterpret medical data, leading to incorrect diagnoses or decisions. While these tools may perform well in controlled laboratory settings, it is unclear how they will perform in real-world scenarios. There are concerns that machine learning algorithms may not be able to account for the myriad of factors that can influence a patient’s health, such as social determinants of health and environmental factors. As such, experts argue that rigorous testing and validation are needed before machine learning tools can be implemented in healthcare settings.

Finally, there is the concern that the use of machine learning tools will lead to a reduction in human interaction and empathy, which are essential for building trust and ensuring patient satisfaction. While machine learning algorithms can provide valuable insights and assist healthcare providers in making more informed decisions, they should not replace human interaction altogether. Patients often require emotional support and reassurance, which can only be provided by a human caregiver. Additionally, the reliance on technology may lead to patients feeling alienated and less involved in their own care, leading to a breakdown in communication and mistrust.

In conclusion, ChatGPT’s near-passing scores on the USLME suggest that AI has the potential to revolutionize the healthcare industry. However, there are limitations and ethical considerations that must be addressed before the widespread adoption of these tools.

No matter what the future holds, we can not deny that society is about to change, and instead of warning about hypochondria by randomly searching the internet for symptoms, we may soon get our medical advice from Doctor GPT or Nurse Bing.