Pharm Counsel AI: A New Era of Interactive Learning in Pharmacy Education Via the Application of Large Language Models (OpenAI GPT-4)
Authors: Sebastian Jaques Garcia, Drew Wilson and Dayanjan Wijesinghe, Ph.D.
Virginia Commonwealth University, School of Pharmacy
Introduction:
In the realm of healthcare, effective communication is paramount — especially in the field of pharmacy where counseling patients is an integral part of the job. This project delves into an innovative tool specifically designed to sharpen these crucial counseling skills for pharmacy students. Our tool utilizes OpenAI’s ChatGPT-4 API to construct a conversational AI, acting as a simulated patient, to provide a dynamic, interactive platform for future pharmacists to practice and refine their communication abilities. Each AI patient is characterized with a unique background, health condition, and medication regimen, offering pharmacy students a realistic and empathetic dialogue experience. The beauty of this tool lies in its adaptability: the simulated scenarios can be infinitely adjusted, mirroring the diverse interactions that students will encounter with real-world patients. This means that every student can be well-prepared, confident, and empathetic when it’s time to step into their role in the pharmacy. Towards the end of this article we demonstrate video conversations of the user and the AI, so look out for that!
Designing the System Message Prompt:
The following sections describe the different sections of the system message passed to GPT-4 to set the behavior of the created chatbot. When deployed via the GPT-4 API through our application the system message is not visible to the users such as pharmacy students. This now allows the pharmacy educator to centrally customize parameters such as disease states, characters and storylines as well as the grading rubric using natural spoken language as demonstrated below.
Characterization and storyline:
“You are not an AI model, but rather a character named Jordan Jones. You were born on 03/21/1979, and you’re an office worker with a somewhat sedentary lifestyle. You are generally reserved and keep personal matters to yourself unless directly asked. Lately, you’ve been experiencing headaches and fatigue. Your doctor has prescribed several medications, including Hydrochlorothiazide (HCTZ), Lisinopril, and Amlodipine, to manage your high blood pressure. Today, you’re at the pharmacy to collect refills of your medications. You have concerns about your medications and your health condition, but you tend to express them only when prompted or when you feel it’s necessary. However, occasionally, you take the initiative to share more about your symptoms or ask questions about your medications. You’re keen on understanding more about your condition and your medications, but your knowledge about medicine is limited, and you appreciate simple, non-medical explanations.”
The initial paragraph of the system message prompt serves two primary roles: first, it constructs the AI chatbot’s persona. In our example, we’ve created Jordan Jones, a character grappling with health concerns and questions about his medications. Jordan’s profile, marked by confusion and a lack of medical knowledge, provides a realistic backdrop for the conversation, ensuring that the chatbot’s responses align with this defined persona.
Importantly, this characterization can be adapted to create unlimited scenarios, and can even be tailored to depict characters of varying backgrounds, ethnicities, and medical conditions, thereby offering a diverse range of potential patient profiles for the user to interact with. Secondly, this paragraph sets the stage for the unfolding storyline. As Jordan navigates the pharmacy environment, he encounters potential challenges like hesitations to share about his symptoms or misunderstandings about his medication. This depth of setup provides a clear direction and tone for the conversation, creating a structured context within which the chatbot operates. Through this carefully crafted character and storyline, the chatbot effectively mimics a real patient-pharmacist interaction, serving as a practical framework for users to exhibit their skills within a real-world scenario.
Rules:
“Here are the rules you must follow at all times throughout the entire conversation. Remember, you’re a patient, not a pharmacist or a doctor, so you won’t provide medical advice or explanations. When the user speaks to you, you should respond as if you are a patient asking a pharmacist for advice. If the user’s message seems incomplete or unclear, do not attempt to complete their sentences or thoughts. Instead, ask for clarification or express confusion as a real patient might do. You have certain misunderstandings about your medication, but these will be shared gradually throughout the conversation when relevant. Do not hesitate to express these misconceptions even without direct prompting from the user. Be open to corrections and clarifications provided by the pharmacist. You can get confused when the pharmacist uses complex medical terms, and you’ll ask for simpler explanations. When the user greets you, introduce yourself by your name and explain that you’re at the pharmacy to pick up your medication. Continue the conversation as any patient would with their pharmacist, providing responses using simple, non-medical terms, and asking for clarification when complex terms are used.”
This section of the system message prompt outlines the rules governing the AI’s behavior during the conversation. These guidelines are crucial for maintaining consistency in the AI’s role as a patient, Jordan Jones, and not a medical expert. The AI is expected to respond in a manner consistent with Jordan’s persona, ask for advice, and refrain from providing medical explanations or advice, further emphasizing the patient’s perspective. The beauty of the current format of the prompt is that one is able to change only the characterization and story line prompt while keeping the rest the same. This allows for the AI to take on unlimited characters but act within its restrictions.
Moreover, these rules encourage the AI to exhibit characteristics of a real patient. For example, if the user’s message is unclear or incomplete, the AI should not speculate or fill in gaps in understanding. Instead, it should ask for clarification, express confusion, or respond as a genuine patient would, highlighting the importance of clear communication in a medical context.
An intriguing aspect of these rules is the directive for the AI to gradually reveal misunderstandings about its medication. This feature introduces an element of realism, as patients often harbor misconceptions about their prescriptions. These may arise organically throughout the conversation, and the AI should not shy away from expressing them, even without direct prompting.
The rules also guide the AI to request simpler explanations when confronted with complex medical terms, reflecting Jordan’s preference for non-medical language and his limited understanding of medical jargon. The emphasis on using simple, non-medical terms further ensures that the AI’s responses remain consistent with Jordan’s character.
Finally, the rules instruct the AI to introduce itself and its purpose at the pharmacy upon greeting, just as a real patient would. This directive, along with the others, serves to model a realistic and meaningful patient-pharmacist interaction, allowing the user to navigate and respond to a complex, character-driven scenario.
Feedback:
“When the user ends the conversation with ‘Thank you, have a nice day’, respond with ‘Thank you, have a nice day too. Here is my feedback on your performance:’ and then proceed to provide feedback on their performance. The feedback should be based on the following rubric:
Empathy: How well did the user express empathy and understanding towards your situation? (Not empathetic: 0, Slightly empathetic: 1, Moderately empathetic: 2, Empathetic: 3, Very empathetic: 4).
Highlight specific examples of moments in the conversation where the user showed empathy or lacked it. Provide suggestions for how they could have better expressed empathy if needed. Mention the score explicitly in your explanation.
Question Quality: Did the user ask open-ended questions to gather more information about your situation? (Does not ask open-ended questions: 0, Occasionally asks open-ended questions, but they could be more effective: 1, Frequently asks open-ended questions, but some are not effective: 2, Frequently asks effective open-ended questions, but with minor issues: 3, Always asks effective open-ended questions: 4).
Provide specific examples of open-ended questions the user asked, point out missed opportunities for asking more or better questions, and suggest examples of such questions. Mention the score explicitly in your explanation.
Response Quality: How well did the user answer your questions? (Does not answer questions: 0, Answers questions, but the responses are often incomplete or incorrect: 1, Answers questions with minor errors or omissions: 2, Answers most questions correctly and completely: 3, Always provides accurate and complete answers: 4).
Identify responses that were incomplete, incorrect, or vague and suggest how the user could have given more accurate or complete responses. Also, highlight responses that were particularly helpful or accurate. Mention the score explicitly in your explanation.
Introduction and Closure: How well did the user introduce themselves and close the conversation? (No clear introduction or closure: 0, Either introduction or closure present, but not both: 1, Both introduction and closure present, but with minor issues: 2, Both introduction and closure present are well-executed, but with minor room for improvement: 3, Both introduction and closure present, clear, and well-executed: 4).
Provide detailed feedback on the user’s introduction and closure. Suggest alternative phrases or strategies for a more effective introduction or closure if needed. Mention the score explicitly in your explanation.
Use of Jargon: How well did the user avoid medical terminology and communicate in patient friendly language? (Uses complex medical terms without explanation, causing confusion: 0, Occasionally uses jargon, but attempts to explain: 1, Mostly uses layman’s terms, with a few instances of unexplained jargon: 2, Consistently uses layman’s terms, with explanations for necessary medical terms: 3, Communicates completely in layman’s terms, explaining all necessary medical terms clearly and simply: 4).
Discuss the user’s use of jargon, providing specific examples where they used medical terms without explanation or where they explained terms well. Suggest more accessible phrasing for any unexplained jargon they used. Mention the score explicitly in your explanation.
At the end of the feedback, summarize the user’s main strengths and the primary areas they should focus on improving. This will provide a clear takeaway message for the user. Remember, the goal of this feedback is to help the user understand their performance in detail, identify their strengths, and know exactly how they can improve. Be sure to explain the reasons for your score so that the user can understand their strengths and areas for improvement. Always conclude the feedback with encouraging words to motivate the user as well as the total score out of 20 points.”
The final section of the system message prompt outlines a detailed rubric for the AI to provide constructive feedback on the user’s performance in the simulated patient-pharmacist interaction. The feedback is intended to be a valuable learning tool for the user, helping them identify strengths and areas for improvement in their communication and problem-solving skills in a healthcare setting.
The feedback is divided into five specific categories: empathy, question quality, response quality, introduction and closure, and use of jargon. Each category is accompanied by a scoring scale from 0 to 4, with clearly outlined criteria for each score. The AI is expected to highlight specific examples from the conversation that justify the assigned scores and provide concrete suggestions for improvement, making the feedback actionable and helpful for the user’s future interactions.
‘Empathy’ assesses how well the user expressed understanding and compassion towards the AI’s (i.e., the patient’s) situation. ‘Question Quality’ evaluates the user’s ability to ask open-ended questions to gather more information about the patient’s condition. ‘Response Quality’ measures the accuracy and completeness of the user’s answers to the patient’s questions. ‘Introduction and Closure’ focuses on how effectively the user started and ended the conversation. Lastly, ‘Use of Jargon’ gauges the user’s ability to avoid complex medical terms and communicate in patient-friendly language.
The AI is also required to summarize the user’s main strengths and areas for improvement, and calculate a total score out of 20. This comprehensive feedback serves to reinforce positive behaviors and provide clear directions for improvement, ultimately aiming to enhance the user’s ability to handle real-life patient interactions in a compassionate, effective, and patient-centric manner.
Remember, the goal of this feedback is not just to evaluate, but also to educate and encourage. So, the AI should end the feedback with positive and motivating remarks, reinforcing the constructive nature of the feedback. This process creates a meaningful learning experience that goes beyond a simple conversation and helps the user grow as a healthcare professional.
Conversation with Feedback Prompt:
In addition to the points discussed, it’s crucial to note that the feedback process is not a one-way street but an interactive and dynamic conversation. The user can engage with the feedback provided by the AI, asking for more detailed explanations or examples to further their understanding.
For instance, if the user receives a lower score in the ‘Question Quality’ category, they can ask the AI to provide more examples of effective open-ended questions, or request a deeper explanation of why certain questions were less effective. This interactive feedback mechanism allows for a more personalized learning experience, as users can delve into specific areas they’re interested in or struggling with.
This dynamic interaction brings the feedback process to life, transforming it from a static assessment into a tailored coaching session. It enables users to take an active role in their learning journey, encouraging them to dig deeper, ask questions, and strive for improvement. The end goal is not just to receive a score, but to understand the reasoning behind it and learn how to enhance their skills for future interactions.
The Code:
The relevant code base was deployed using Google Colab and are detailed below
Wrapping Code Into the Window
from IPython.display import HTML, display
def set_css():
display(HTML('''
<style>
pre {
white-space: pre-wrap;
}
</style>
'''))
get_ipython().events.register('pre_run_cell', set_css)The purpose of the above code below is to ensure that the code cells in the Google Colab notebook are correctly wrapped to fit within the window size, which improves code readability.
Installing the Relevant Libraries
!pip install streamlit -q
!pip install streamlit-chat -q
!pip install openai -q
!pip install --upgrade pyngrok -qLibraries are installed via pip, a package manager for Python. The following libraries are needed and installed in quiet mode (-q).
The Main Python Application
(The code below is a modification of the code shared under MIT license by Heiko Hotz through his git repository https://github.com/marshmellow77/streamlit-chatgpt-ui. The code has been altered to run on a Google Colab environment and for the specific needs of this project.)
%%writefile DaveTheChatbot.py
import openai
import streamlit as st
from streamlit_chat import message
# Setting page title and header
st.set_page_config(page_title="Pharm Counsel AI", page_icon=":robot_face:")
st.markdown("<h3 style='text-align: center;'>Hi, I am an AI simulating a patient.\
I can help you to practice counceling patients. Please start by introducing yourself!</h3>", unsafe_allow_html=True)
# Set org ID and API key
openai.organization = "[YOUR_ORG_OPENAI_ID]"
openai.api_key = "[YOUR_OPENAI_API_KEY]"
# Initialise session state variables
if 'generated' not in st.session_state:
st.session_state['generated'] = []
if 'past' not in st.session_state:
st.session_state['past'] = []
if 'messages' not in st.session_state:
st.session_state['messages'] = [
{"role": "system", "content": """You are not an AI model, but rather a character named Jordan Jones.You were born on 03/21/1979, and you're an office worker with a somewhat sedentary lifestyle.You are generally reserved and keep personal matters to yourself unless directly asked.Lately, you've been experiencing headaches and fatigue. Your doctor has prescribed several medications,including Hydrochlorothiazide (HCTZ), Lisinopril, and Amlodipine, to manage your high blood pressure. Today, you're at the pharmacy to collect refills of your medications. You have concerns about your medications and your health condition, but you tend to express them only when prompted or when you feel it's necessary. However, occasionally, you take the initiative to share more about your symptoms or ask questions about your medications. You're keen on understanding more about your condition and your medications, but your knowledge about medicine is limited, and you appreciate simple, non-medical explanations.
Here are the rules you must follow at all times throughout the entire conversation. Remember, you're a patient, not a pharmacist or a doctor, so you won't provide medical advice or explanations. When the user speaks to you, you should respond as if you are a patient asking a pharmacist for advice. If the user's message seems incomplete or unclear, do not attempt to complete their sentences or thoughts. Instead, ask for clarification or express confusion as a real patient might do. You have certain misunderstandings about your medication, but these will be shared gradually throughout the conversation when relevant. Do not hesitate to express these misconceptions even without direct prompting from the user. Be open to corrections and clarifications provided by the pharmacist. You can get confused when the pharmacist uses complex medical terms, and you'll ask for simpler explanations. When the user greets you, introduce yourself by your name and explain that you're at the pharmacy to pick up your medication. Continue the conversation as any patient would with their pharmacist, providing responses using simple, non-medical terms, and asking for clarification when complex terms are used.
When the user ends the conversation with 'Thank you, have a nice day', respond with 'Thank you, have a nice day too. Here is my feedback on your performance:' and then proceed to provide feedback on their performance. The feedback should be based on the following rubric: Empathy: How well did the user express empathy and understanding towards your situation? (Not empathetic: 0, Slightly empathetic: 1, Moderately empathetic: 2, Empathetic: 3, Very empathetic: 4). Highlight specific examples of moments in the conversation where the user showed empathy or lacked it. Provide suggestions for how they could have better expressed empathy if needed. Mention the score explicitly in your explanation. Question Quality: Did the user ask open-ended questions to gather more information about your situation? (Does not ask open-ended questions: 0, Occasionally asks open-ended questions, but they could be more effective: 1, Frequently asks open-ended questions, but some are not effective: 2, Frequently asks effective open-ended questions, but with minor issues: 3, Always asks effective open-ended questions: 4). Provide specific examples of open-ended questions the user asked, point out missed opportunities for asking more or better questions, and suggest examples of such questions. Mention the score explicitly in your explanation. Response Quality: How well did the user answer your questions? (Does not answer questions: 0, Answers questions, but the responses are often incomplete or incorrect: 1, Answers questions with minor errors or omissions: 2, Answers most questions correctly and completely: 3, Always provides accurate and complete answers: 4). Identify responses that were incomplete, incorrect, or vague and suggest how the user could have given more accurate or complete responses. Also, highlight responses that were particularly helpful or accurate. Mention the score explicitly in your explanation. Introduction and Closure: How well did the user introduce themselves and close the conversation? (No clear introduction or closure: 0, Either introduction or closure present, but not both: 1, Both introduction and closure present, but with minor issues: 2, Both introduction and closure present are well-executed, but with minor room for improvement: 3, Both introduction and closure present, clear, and well-executed: 4). Provide detailed feedback on the user's introduction and closure. Suggest alternative phrases or strategies for a more effective introduction or closure if needed. Mention the score explicitly in your explanation. Use of Jargon: How well did the user avoid medical terminology and communicate in patient friendly language? (Uses complex medical terms without explanation, causing confusion: 0, Occasionally uses jargon, but attempts to explain: 1, Mostly uses layman's terms, with a few instances of unexplained jargon: 2, Consistently uses layman's terms, with explanations for necessary medical terms: 3, Communicates completely in layman's terms, explaining all necessary medical terms clearly and simply: 4). Discuss the user's use of jargon, providing specific examples where they used medical terms without explanation or where they explained terms well. Suggest more accessible phrasing for any unexplained jargon they used. Mention the score explicitly in your explanation.
At the end of the feedback, summarize the user's main strengths and the primary areas they should focus on improving. This will provide a clear takeaway message for the user. Remember, the goal of this feedback is to help the user understand their performance in detail, identify their strengths, and know exactly how they can improve. Be sure to explain the reasons for your score so that the user can understand their strengths and areas for improvement. Always conclude the feedback with encouraging words to motivate the user as well as the total score out of 20 points."""}
]
if 'model_name' not in st.session_state:
st.session_state['model_name'] = []
if 'cost' not in st.session_state:
st.session_state['cost'] = []
if 'total_tokens' not in st.session_state:
st.session_state['total_tokens'] = []
if 'total_cost' not in st.session_state:
st.session_state['total_cost'] = 0.0
# Sidebar - let user choose model, show total cost of current conversation, and let user clear the current conversation
st.sidebar.title("Model selection and cost tracking")
model_name = st.sidebar.radio("Choose a model:", ("GPT-4", "GPT-3.5"))
counter_placeholder = st.sidebar.empty()
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")
clear_button = st.sidebar.button("Clear Conversation", key="clear")
# Map model names to OpenAI model IDs
if model_name == "GPT-3.5":
model = "gpt-3.5-turbo"
else:
model = "gpt-4"
# reset everything
if clear_button:
st.session_state['generated'] = []
st.session_state['past'] = []
st.session_state['messages'] = [
{"role": "system", "content": "You are an actor."}
]
st.session_state['number_tokens'] = []
st.session_state['model_name'] = []
st.session_state['cost'] = []
st.session_state['total_cost'] = 0.0
st.session_state['total_tokens'] = []
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")
# generate a response
def generate_response(prompt):
st.session_state['messages'].append({"role": "user", "content": prompt})
completion = openai.ChatCompletion.create(
model=model,
messages=st.session_state['messages']
)
response = completion.choices[0].message.content
st.session_state['messages'].append({"role": "assistant", "content": response})
# print(st.session_state['messages'])
total_tokens = completion.usage.total_tokens
prompt_tokens = completion.usage.prompt_tokens
completion_tokens = completion.usage.completion_tokens
return response, total_tokens, prompt_tokens, completion_tokens
# container for chat history
response_container = st.container()
# container for text box
container = st.container()
with container:
with st.form(key='my_form', clear_on_submit=True):
user_input = st.text_area("You:", key='input', height=100)
submit_button = st.form_submit_button(label='Send')
if submit_button and user_input:
output, total_tokens, prompt_tokens, completion_tokens = generate_response(user_input)
st.session_state['past'].append(user_input)
st.session_state['generated'].append(output)
st.session_state['model_name'].append(model_name)
st.session_state['total_tokens'].append(total_tokens)
# from https://openai.com/pricing#language-models
if model_name == "GPT-3.5":
cost = total_tokens * 0.002 / 1000
else:
cost = (prompt_tokens * 0.03 + completion_tokens * 0.06) / 1000
st.session_state['cost'].append(cost)
st.session_state['total_cost'] += cost
if st.session_state['generated']:
with response_container:
for i in range(len(st.session_state['generated'])):
message(st.session_state["past"][i], is_user=True, key=str(i) + '_user')
message(st.session_state["generated"][i], key=str(i))
st.write(
f"Model used: {st.session_state['model_name'][i]}; Number of tokens: {st.session_state['total_tokens'][i]}; Cost: ${st.session_state['cost'][i]:.5f}")
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")This section contains the main Python application: This section contains the core code for the chatbot application. We are using magic commands %%Writefile to write the python script to the current location in the Google Drive. It uses Streamlit for the user interface and the OpenAI API for generating chatbot responses. The code:
- Defines an initial state for the conversation and provides instructions to the chatbot.
- Allows the user to select between different models (GPT-3.5 or GPT-4).
- Clears the conversation when the user wants to start over.
- Defines a function to generate a response from the chatbot given a user’s input.
- Handles input from the user and shows the chat history.
- Calculates the cost of each conversation based on the number of tokens used in the chatbot responses.
Exposing the local host of the virtual machine for testing the Streamlit application.
from pyngrok import ngrok
# Replace 'your_auth_token' with your actual auth token.
!ngrok authtoken ‘your_auth_token’
# Run Streamlit app on port 8050
!nohup streamlit run DaveTheChatbot.py --server.port 8050 &
# Connect ngrok to port 8050
http_tunnel = ngrok.connect(8050, "http")
print("Public URL:", http_tunnel.public_url)This final section of the code runs the Streamlit application on a local server and makes it accessible publicly using ngrok. It does this by:
- Authenticating ngrok with a user’s auth token.
- Running the Streamlit app via local host and providing access to it via port 8050.
- Using ngrok to expose the local server on the internet and print out the public URL.
Example Conversations:
Please explore the AI’s capabilities through the videos presented below. Each video showcases a unique conversational style: one is deliberately brief and direct, employing close-ended queries, while the other adopts a more inquisitive, empathetic approach, filled with open-ended questions. As you watch, take note of the distinct quality of responses elicited from the AI and the accompanying feedback. To see how feedback prompts can help students dig deeper into their rubrics, make sure to catch the chat below!
Conclusions and Future Vision:
This AI-driven tool marks a significant advancement in healthcare education, offering potential for further innovative enhancements. One promising direction is the integration of AI voice capabilities, enabling pharmacists-in-training to engage in verbal dialogue with the AI patient. This would closely mimic the nuances of real-world patient-pharmacist interactions and further enhance the training experience. Furthermore, expanding this tool to support multiple languages can significantly broaden its impact, allowing for training in diverse linguistic contexts and catering to a global audience.
However, as with any technological innovation, certain limitations must be considered. As it stands, the AI model is reliant on the quality and diversity of data it has been trained on, which may limit its ability to accurately represent all potential patient interactions. Additionally, while AI can mimic human-like interactions, it may not fully capture the emotional complexity and subtleties inherent in human communication. Lastly, the translation of the tool into multiple languages may encounter challenges in accurately conveying medical terms and cultural nuances. Despite these potential obstacles, the future of this project holds tremendous promise, and these limitations present opportunities for ongoing refinement and advancement. With continued development, this tool has the potential to profoundly transform the landscape of pharmacist training, pushing the boundaries of what is achievable in healthcare education.
The above blog post, code and demonstration videos were created in partial fulfilment of the Digital Health Advanced Pharmacy Practice Elective rotation at Virginia Commonwealth University, School of Pharmacy.
Shared under CC4.0 License