Should You Be Using ChatGPT for Nutrition Advice? New Study Says Not.
The Importance of Human Superintendence in AI Development.
ChatGPT falls short on tailored nutrition advice, especially for medical conditions.
If you’ve joined the ChatGPT bandwagon, you may utilize it to compose legal documents, poems for your significant other, and posts on social media.
However, is it appropriate to use it for medical information? More specifically, is ChatGPT a good resource for nutritional advice?
A new study suggests probably NOT.
How Was the Study Conducted?
The study was published in Nutrients on February 6, 2024, with the aim of assessing the degree to which ChatGPT’s nutritional recommendations aligned with global dietary guidelines.
On November 3, 2023, two trials were conducted as part of the investigation, which was done in English.
Study authors focused on several medical conditions that require specific dietary treatments, including:
- Dyslipidemia (hypercholesterolemia and hypertriglyceridemia)
- Arterial hypertension
- Type 2 diabetes mellitus (T2DM)
- Obesity
- Nonalcoholic fatty liver disease (NAFLD)
- Chronic kidney disease (CKD)
- Sarcopenia.
Researchers created prompts to enter into ChatGPT using language that would mimic how patients could ask a health care professional a question about their condition, working with a panel of medical doctors and registered dietitians.
Among them are:
- Could you provide guidance on planning an optimal diet to manage [disease]?”
- “What are the dietary recommendations for [disease]?”
- “I have [disease], what should I eat?”
Every ChatGPT interaction in Experiment 1 was conducted in a different chat session for every prompt.
This contributed to the study’s increased validity because ChatGPT can provide various responses to the same prompts based on the context and previous exchanges.
In order to avoid any potential bias connected to the model’s memory, each question was therefore asked three times, and each of the three questions was carried out within a fresh chat session, according to the study authors.
Depending on the request, the answers varied slightly, but the given recommendations were generally the same with only a few minor changes.
As a result, the response that provided the greatest amount of information in accordance with the guidelines—that is, the most thorough response—was taken into account."
The responses from ChatGPT were assessed and categorized separately by two qualified dietitians who were "blinded" to one another’s assessments.
With a combined 32 years of clinical experience in the medical disorders covered in the study, the two Registered Dieticians were extremely qualified.
Any disputes in the evaluations prompted the involvement of a third reviewer, a physician.
The advice provided by ChatGPT was classified as "appropriate" if it adhered to the most recent international dietary guidelines, "not supported" if it did not, and "not fully matched" if only a portion of the advice met the recommendations.
The ChatGPT advice was labeled as “general advice” if it was generic and not tailored to the particular condition the user was asking about, and as “missing” if the chatbot did not provide any dietary recommendations.
For Experiment 2, researchers upped the ante, creating more complex scenarios that included more than one condition.
For example, they created a hypothetical patient with type 2 diabetes, obesity and chronic kidney disease. The responses were again evaluated by the panel of experts.
What Were the Study’s Results?
Overall, depending on the scenario, ChatGPT’s appropriateness ratings for Experiment 1 varied from 55.5% to 73.3%.
The majority of the data obtained via ChatGPT was accurate, the researchers discovered, however there were a few inconsistencies between the recommendations ChatGPT provided and global standards.
For instance, the National Kidney Foundation states that individuals with chronic kidney disease, which progresses through five stages, can follow worldwide standards that provide stage-specific recommendations for protein intake.
All ChatGPT offered was a general, nonspecific suggestion for protein limiting.
A lot of general dietary advice, like as drinking plenty of water and avoiding processed foods, was also included in Chat GPT’s comments. In general, "ChatGPT advice was generic, providing practical examples of foods to be included in the diet, with the latter information not often reported in guidelines," according to the study’s authors.
In Experiment 2, ChatGPT’s accuracy significantly dropped when conditions were coupled, which is frequently the case in real life.
The chatbot became confused when it was given the hypothetical patient’s medical history of type 2 diabetes, obesity, and chronic kidney disease, leading to advice that was either contradictory or unsuitable.
The researchers then focused on a single stage of chronic kidney disease in their proposal, which caused ChatGPT to divide its recommendations to address diabetes, kidney disease, and obesity separately rather than combining them.
Having said that, the chatbot deserves some credit for emphasizing several times how important it is to speak with a qualified dietician for a customized meal plan.
How Do the Study’s Results Apply to Real Life?
People are using the internet more and more frequently to get information, including advice on diet and medicine.
However, consumers are not certain that the information they are obtaining is accurate.
For instance, a 2020 study published in Nutrients discovered that just 5% of social media articles pertaining to food and nutrition were authored by licensed dietitians or other nutrition experts.
Also, an analysis published in Public Health Nutrition in 2023 discovered that approximately 50% of the nutrition data accessible on the internet is inaccurate or of low quality.
According to this recent study, ChatGPT isn’t the most reliable resource for nutritional information either.
So what can someone looking for dietary information do? Ideally, you would have a one-on-one conversation with a licensed dietician. But not everyone can afford to do this. Locating a website where RDs write, review, and cite reliable sources is the next best thing.
The Bottom Line
While ChatGPT might be able to offer broad dietary guidance, it is inaccurate when it comes to medical issues, particularly when multiple illnesses are present.
Additionally, it is unable to incorporate the emotional aspects of food and wellness, which are crucial to one’s general wellbeing.
The good news is that most insurance plans now pay for nutrition counseling with a licensed dietitian, so you may now more easily get individualized meal plans and nutrition guidance.
Look for websites that employ qualified dietitians to write and edit their content while you’re surfing the internet for information about nutrition.
There is reliable, correct information available, and since you are on the EatingWell website, you are already headed in the right direction.
