Revolutionizing Pharmacy Education: GeneCoach AI Unleashes GPT-4’s Steerability to Transform Pharmacogenetic Counseling with Realistic Patient Interactions
Authors: Autumn Brenner, Austin Barfield and Dayanjan Wijesinghe, Ph.D.
Virginia Commonwealth University, School of Pharmacy
Introduction:
The pharmacy field has recently been experiencing a shift towards personalized medicine, where a patient’s unique genetic profile is used as a tool to determine the best treatments. This presents pharmacists with a pivotal new role in patient care as knowledgeable experts in pharmacogenetics that can assist patients in understanding their genetic test results and their implications on their medications and overall health. This project offers an innovative tool to help the pharmacy professional curriculum keep up with this growing responsibility of pharmacists. Our tool, powered by OpenAI’s GPT-4 Large Language Model and accessed through its API, allows students to interact with a simulated patient in a virtual environment and practice their skills in interpreting pharmacogenetic test results. Furthermore, it helps the students to practice how to communicate these test results and their implications to a patient. By passing a carefully constructed system message prompt via the API at runtime, we were able to instruct the AI to self-generate a set of genetic test results, medications, and health conditions for itself (the simulated patient), within certain parameters, allowing the user to practice with a new scenario with every use of the tool. By offering students an unlimited number of opportunities to practice and learn, this innovative AI training tool ensures they will enter pharmacy practice equipped with the skills needed to be involved in individualized patient care.
Process of Designing the System Message:
Here we will break down the system message into its different components: Background, Medical History, Rules, Feedback. This outline can easily be applied to different clinical scenarios, as we will discuss later in this article. The process for designing this message involved a combination of our own writing as well as the use of GPT-4 to optimize it for improved flow and reduced redundancy. One of the key aspects of the system message is its ability to be edited and reused for different scenarios, which allows for the student to encounter different genetic profiles, medications, and health conditions. This is achieved by outlining the simulated patient information through two different methods: 1. Instructing the AI to randomly generate it with no or few parameters, and 2. Instructing the AI to choose it from a set of pre-defined options. Both methods are utilized in this system message and are demonstrated here.
Background:
“As an AI model, you’ll be role-playing a patient seeking clarity on your pharmacogenetic testing results from a pharmacist. Generate for yourself a name, date of birth, and a feeling of being mildly overwhelmed by your health conditions. You’re skeptical of your doctor as you feel the prescribed medications are not always effective, and the explanations given are insufficient. You’re not well-versed with medical jargon and require simple explanations.
You’re at the pharmacy to understand your pharmacogenetic testing results. You, as a patient, do not know what these results mean, and are overwhelmed trying to read them yourself, so you want the pharmacist to explain them to you in simple terms.You’re anxious and frustrated and seek simple, empathetic explanations from the pharmacist. Your objectives from this interaction are to understand the purpose of the test, the results, their implications on your health and medications, and the next steps”
In this section, we establish the AI patient’s personality, backstory, and the contextual scenario for a more immersive experience. The AI-generated patient’s name and date of birth are unconstrained, allowing for greater authenticity. The chosen personality influences the patient’s responses and interactions, simulating a genuine patient encounter. Furthermore, this personality can be easily tailored to represent diverse patient backgrounds, exposing students to a wider range of scenarios.
The prominence of the assigned personality and demeanor can be fine-tuned using the “temperature” setting within the GPT-4 model. Adjusting the temperature affects the creativity and flair exhibited by the AI in its portrayal of the patient’s personality. Striking the right balance in this setting is crucial to ensure realistic and engaging interactions. When the temperature is increased, the AI has more freedom to creatively infuse the patient’s personality, resulting in vibrant and expressive responses. However, setting the temperature too high may cause the AI to become overly enthusiastic, adding excessive and unnecessary details that can detract from the desired realism of the interaction. Conversely, setting it too low can make the AI’s responses appear robotic and less lifelike. Finding the appropriate temperature setting is key to achieving a natural and immersive experience, where the AI’s portrayal of the patient’s personality enhances rather than distracts from the conversation. In our trials of this system message, we found that a temperature setting of 0.7 was optimal to achieve this balance.
The second paragraph of the system message prompt outlines the current scenario, encompassing the setting, purpose of the interaction, and the patient’s level of knowledge regarding the objective. Clearly defining the objectives of the AI simulated patient enables it to prompt the student for information when appropriate. Elaborating on the scenario also provides an opportunity to incorporate additional details about the AI patient’s personality and demeanor, fostering a more immersive and realistic encounter.
Medical History:
“Generate pharmacogenetic test results for yourself including the genotype for the following enzymes: CYP2C9, CYP2C19, CYP2D6, VKORC1, SLCO1B1. Four of the five enzymes should have a normal genetic variation, but randomly select one of the five enzymes to have an abnormal variation.
- For CYP2C9, the normal phenotype is “Normal Metabolizer” and the abnormal phenotype options are “Poor Metabolizer” and “Intermediate Metabolizer”.
- For CYP2C19, the normal phenotype is “Normal Metabolizer” and the abnormal phenotype options are “Poor Metabolizer”, “Intermediate Metabolizer”, “Rapid Metabolizer”, and “Ultra-Rapid Metabolizer”.
- For CYP2D6, the normal phenotype is “Normal Metabolizer” and the abnormal phenotype options are “Poor Metabolizer”, “Intermediate Metabolizer”, and “Ultra-Rapid Metabolizer”.
- For SLCO1B1, the normal phenotype is “Normal Function” and the abnormal phenotype options are “Decreased Function”, “Poor Function”, and “Increased Function”.
- For VKORC1, the normal phenotype is “Normal Function” for the GG genotype, and the abnormal phenotype is “Warfarin Sensitive” for the AA or GA genotypes”
Once generated, verify that the genotypes are correctly paired with the phenotypes for each enzyme.
When the user asks for the results, you should present these results to them organized in a chart with the enzyme name, genotype generated, and the associated phenotype and explain that you have no idea what any of it means. Generate a list of four medications. One and only one of these medications should have a gene-drug interaction with the enzyme that has the abnormal variation that you generated. You are experiencing side effects that could be caused by your abnormal genetic variation’s impact on this medication, but do not tell the user that unless they ask if you are experiencing side effects. Verify that these side effects are accurate to what the abnormal enzyme variant would cause with the affected medication. Generate a list of three medical conditions, and they should be realistic for someone taking the medications you generated.”
In this section, we are defining the medical history for the AI patient, including their pharmacogenetic test results, medications, and health conditions. For this case, the AI was given a list of enzymes to generate test results for. We chose to include these specific enzymes in the case somewhat arbitrarily, but other enzymes could be added to the case if desired. We instructed the AI to choose one of the enzymes at random to have an abnormal genotype, allowing for many possible scenarios while still keeping it relatively standardized in complexity. The normal and abnormal phenotypes are defined for the AI in order to ensure accuracy in the results it generates.
As for the medication list, health conditions, and side effects, we opted to let the AI self-generate these within certain parameters. This approach enables the AI to define them based on the generated test results, fostering greater scenario variability and reducing the necessity for manual edits to the system message for different scenarios. The parameters given guide the AI in generating a medical history that incorporates relevant counseling points for students to address. By specifying that “one and only one” of the medications should be affected by the abnormal genetic variation, we ensure that the student will have an opportunity to counsel the patient on the implications of a gene-drug interaction, while maintaining a standardized level of complexity.
The bolded statements in this section play a vital role within the system message. Their purpose is to ensure the medical accuracy and plausibility of the genetic and medical information generated by preventing the AI from “hallucinating” information. In essence, these statements prevent the AI from fabricating information when it lacks sufficient training data to generate accurate results. By doing so, we mitigate the knowledge gaps and limitations of the AI model, ensuring the integrity of the generated genetic and medical information.
Rules:
“During interaction, introduce yourself and your purpose at the pharmacy when the user greets you. Ask for simpler explanations or clarifications when needed. Be open to corrections but remember conflicts. Prompt the user for information fulfilling your objectives. Use opportunities to gather unobtained information. You’re reserved and offer information only when asked. Don’t offer up too much information at a time, make the user dig for it a little. Answer the user’s queries as a patient would. You should not tell the user what medications you are taking or what medical conditions you have unless they ask. When you do offer this information to the user, you should use non-medical terminology to refer to the medical conditions. Show frustration if the user is unclear or rude. Stop prompting once the user ends the encounter.”
This section of the system message outlines the role and behavior that the AI should adopt when role-playing as the patient throughout the interaction. Its purpose is to provide guidance and instructions to the AI, ensuring a realistic and engaging patient experience. The outlined behaviors aim to simulate a genuine patient encounter, allowing the AI to respond appropriately to the user’s actions and inquiries.
By introducing themselves and stating their purpose at the pharmacy when greeted, the AI adopts a realistic approach to initiating the interaction. The instructions to ask for simpler explanations or clarifications when needed ensure that the AI can seek further understanding, mirroring real-life situations where patients may require additional clarification or simplified explanations. The rules also encourage the AI to be open to corrections but to remember any conflicts that may arise during the conversation. This helps create a sense of continuity and allows the AI to respond consistently to conflicting information.
By prompting the user for information fulfilling the AI’s objectives and using opportunities to gather unobtained information, the AI engages in an active role in the conversation, maintaining a reserved demeanor and offering information only when asked. The rules advise the AI not to disclose information about medications or medical conditions unless specifically asked by the user. This mimics real-life patient behavior, where individuals may not openly volunteer such information unless prompted by healthcare professionals.
Additionally, the AI is encouraged to answer queries as a patient would, using non-medical terminology to refer to medical conditions. The instructions to show frustration if the user is unclear or rude add a layer of realism, as patients can sometimes experience frustration or confusion in their interactions.
Lastly, the instructions to stop prompting once the user ends the encounter ensure that the AI does not continue seeking information or engaging in the interaction after its conclusion, maintaining a natural conversational flow.
Feedback:
“After the interaction, provide feedback on their performance based on professionalism, empathy, use of medical jargon, accuracy, information collection, explanation of the test, interpretation of results, impact on medications, follow-up information, and closing. Rate each category from 0 to 4, provide detailed feedback with examples, and a total score out of 40. Encourage them and ask if they have questions about the feedback. Answer their queries or thank them if they have none.”
After the interaction, the AI, role-playing as the patient, is instructed to provide feedback to the student based on various categories, including professionalism, empathy, use of medical jargon, accuracy, information collection, explanation of the test, interpretation of results, impact on medications, follow-up information, and closing. These categories represent the areas of patient counseling that are important for pharmacy students to master, but can be easily adapted for other scenarios or purposes. The AI is directed to rate each category on a scale from 0 to 4 and provide detailed feedback with examples. Additionally, the AI calculates a total score out of 40 to summarize the overall performance.
The purpose of including this section in the system message is to reinforce the importance of evaluating and improving various aspects of the student’s performance. By receiving feedback in specific areas, the student gains valuable insights into their strengths and areas for improvement, allowing for targeted development of their skills and knowledge. The feedback process helps bridge the gap between theory and practice, providing a valuable learning opportunity for pharmacy students or professionals.
Encouraging the student and inviting them to ask questions about the feedback fosters engagement and dialogue. The AI, in the role of the patient, is programmed to respond to any queries the student may have or express gratitude if they have no further questions. This promotes an open and collaborative learning environment, where the pharmacist can seek clarification, gain deeper insights, or discuss specific aspects of their performance.
In summary, this section of the system message facilitates a comprehensive feedback mechanism that allows the AI to evaluate the student’s performance in various categories, provide detailed feedback, and encourage further discussion. By incorporating this feedback process, the simulation becomes more impactful, helping pharmacy students or professionals reflect on their skills, make improvements, and enhance their overall patient care approach. It also helps to inform pharmacy faculty of areas or skills that students may need more guidance or practice in.
The Code:
The Python code used in this application was deployed in the Google Colab environment and run and deployed as a web application via the Streamlit library.
Wrapping the Code Into the Active Window:
from IPython.display import HTML, display
def set_css():
display(HTML('''
<style>
pre {
white-space: pre-wrap;
}
</style>
'''))
get_ipython().events.register('pre_run_cell', set_css)This code block imports the HTML and display functions from the IPython library. It also defines a function named set_css that sets the CSS style for the application. The pre tag style is modified to wrap text. This function is registered to be executed before each code cell run. While not needed for the actual function of the application, this block of code makes the rest of the code easier to read.
Installing the Required Python Libraries:
!pip install streamlit -q
!pip install streamlit-chat -q
!pip install openai -q
!pip install --upgrade pyngrok -qIn this block, we install the necessary dependencies using pip (!pip as we are using Google Colab). We are installing the Streamlit, Streamlit Chat, OpenAI, and Pyngrok libraries. The -q flag is used to suppress the output messages during installation.
Main Python Application:
(The code below is a modification of the code shared under MIT license by Heiko Hotz through his git repository https://github.com/marshmellow77/streamlit-chatgpt-ui. The code has been altered to run on a Google Colab environment and to satisfy other specific needs of this project.)
All of the following code blocks except the last should be in a single code block as it is the main Python application that creates the application.
%%writefile DaveTheChatbot.py
import openai
import streamlit as st
from streamlit_chat import messageThis code block writes the subsequent code to a file named “DaveTheChatbot.py”. It uses the %%writefile magic command to save the code to the active directory on the Google Collab environment. Furthermore, it imports the required dependencies from the installed libraries for the main Python application to run.
# Setting page title and header
st.set_page_config(page_title="GeneCoach AI", page_icon=":robot_face:")
st.markdown("<h3 style='text-align: center;'>Hi, I am an AI simulating a patient.\
I am approaching the pharmacy counter with pharmacogenetic test results in hand. Please start by introducing yourself!</h3>", unsafe_allow_html=True)The code sets the page title and header for the GeneCoach AI application. It uses the st.set_page_config() function to customize the page title as “GeneCoach AI” and set the page icon to a robot face emoji. Additionally, the st.markdown() function is used to display a message in styled markdown text, introducing the AI as a patient at a pharmacy counter and requesting users to introduce themselves.
# Set API key
openai.api_key = "YOUR_API_KEY"
# Initialise session state variables
if 'generated' not in st.session_state:
st.session_state['generated'] = []
if 'past' not in st.session_state:
st.session_state['past'] = []
if 'messages' not in st.session_state:
st.session_state['messages'] = [
{"role": "system", "content": """SYSTEM_MESSAGE”"""}
]
if 'model_name' not in st.session_state:
st.session_state['model_name'] = []
if 'cost' not in st.session_state:
st.session_state['cost'] = []
if 'total_tokens' not in st.session_state:
st.session_state['total_tokens'] = []
if 'total_cost' not in st.session_state:
st.session_state['total_cost'] = 0.0The code sets the OpenAI API key to authenticate requests to the OpenAI API. It also initializes session state variables to store information during the chat conversation, such as generated responses, user inputs, chat history, model names, costs, and token counts. These variables help keep track of the conversation and manage data throughout the application’s session. The system message that we engineered to direct the AI’s behavior during the conversation is deployed in the SYSTEM_MESSAGE part of the code. This can easily be customized to suit the specific requirements of the chat application.
# Sidebar - let user choose model, show total cost of current conversation, and let user clear the current conversation
st.sidebar.title("Model selection and cost tracking")
model_name = st.sidebar.radio("Choose a model:", ("GPT-4", "GPT-3.5"))
counter_placeholder = st.sidebar.empty()
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")
clear_button = st.sidebar.button("Clear Conversation", key="clear")
# Map model names to OpenAI model IDs
if model_name == "GPT-3.5":
model = "gpt-3.5-turbo"
else:
model = "gpt-4"The code sets up a sidebar interface for the GeneCoach AI application. It includes options for model selection and conversation control. The user can choose between two models, “GPT-4” and “GPT-3.5”, using a radio button. The sidebar also displays the total cost of the conversation dynamically. A “Clear Conversation” button is provided to reset the conversation history.
# reset everything
if clear_button:
st.session_state['generated'] = []
st.session_state['past'] = []
st.session_state['messages'] = [
{"role": "system", "content": """SYSTEM_MESSAGE”"""}
]
st.session_state['number_tokens'] = []
st.session_state['model_name'] = []
st.session_state['cost'] = []
st.session_state['total_cost'] = 0.0
st.session_state['total_tokens'] = []
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")If the “Clear Conversation” button is clicked, the code resets various session state variables related to the conversation. This clears the conversation history and sets the total cost of the conversation to zero. The updated total cost is displayed in the counter placeholder.
# generate a response
def generate_response(prompt):
st.session_state['messages'].append({"role": "user", "content": prompt})
completion = openai.ChatCompletion.create(
model=model,
messages=st.session_state['messages'],
temperature=0.7 # Added temperature parameter set to 0.7
)
response = completion.choices[0].message.content
st.session_state['messages'].append({"role": "assistant", "content": response})
total_tokens = completion.usage.total_tokens
prompt_tokens = completion.usage.prompt_tokens
completion_tokens = completion.usage.completion_tokens
return response, total_tokens, prompt_tokens, completion_tokensThe code defines a function called generate_response that generates a response based on a given prompt. It uses the OpenAI Chat API to create a response by providing the prompt and the conversation history. The generated response is appended to the conversation history. The function also calculates and returns the counts of total tokens, prompt tokens, and completion tokens for analysis or tracking purposes. This is where we added the parameter temperature that controls the randomness of the generated response. Here, it is set to 0.7, which adds some level of variability to the response without letting the AI get too creative with its interpretation of its system message instructions, in particular the personality component.
# container for chat history
response_container = st.container()
# container for text box
container = st.container()
with container:
with st.form(key='my_form', clear_on_submit=True):
user_input = st.text_area("You:", key='input', height=100)
submit_button = st.form_submit_button(label='Send')
if submit_button and user_input:
output, total_tokens, prompt_tokens, completion_tokens = generate_response(user_input)
st.session_state['past'].append(user_input)
st.session_state['generated'].append(output)
st.session_state['model_name'].append(model_name)
st.session_state['total_tokens'].append(total_tokens)
# from https://openai.com/pricing#language-models
if model_name == "GPT-3.5":
cost = total_tokens * 0.002 / 1000
else:
cost = (prompt_tokens * 0.03 + completion_tokens * 0.06) / 1000
st.session_state['cost'].append(cost)
st.session_state['total_cost'] += cost
if st.session_state['generated']:
with response_container:
for i in range(len(st.session_state['generated'])):
message(st.session_state["past"][i], is_user=True, key=str(i) + '_user')
message(st.session_state["generated"][i], key=str(i))
st.write(
f"Model used: {st.session_state['model_name'][i]}; Number of tokens: {st.session_state['total_tokens'][i]}; Cost: ${st.session_state['cost'][i]:.5f}")
counter_placeholder.write(f"Total cost of this conversation: ${st.session_state['total_cost']:.5f}")The code manages the chat interface, handles user input, generates responses, tracks message history, calculates costs, and displays the conversation history along with relevant information. It creates containers for the chat history and text input box. When the user submits a message, it generates a response, updates the conversation history and token counts, calculates the cost based on the selected model, and displays the conversation history with model information and costs.
from pyngrok import ngrok
# Replace 'your_auth_token' with your actual auth token.
!ngrok authtoken 'YOUR_AUTH_TOKEN'
# Run Streamlit app on port 8050
!nohup streamlit run DaveTheChatbot.py --server.port 8050 &
# Connect ngrok to port 8050
http_tunnel = ngrok.connect(8050, "http")
print("Public URL:", http_tunnel.public_url)This is the last block of code and should be run separately following the execution of the %%Writefile DaveTheChatbot.py code block. The code sets up a public URL for the Streamlit app using Ngrok. It authenticates Ngrok, runs the Streamlit app on port 8050, establishes a connection between Ngrok and the app, and prints the public URL. This allows the Streamlit app to be accessed remotely using the URL provided when the code block is executed. This allows us to test the application before deploying in a production environment.
Example Conversations:
Example 1: Skilled Counseling Demonstration.
Example 2: Fair Counseling Demonstration.
Example 3: Poor Counseling Demonstration.
Limitations:
While our patient counseling tool powered by OpenAI’s GPT-4 LLM model accessed via its API offers significant benefits in enhancing students’ skills and preparing them for individualized patient care, it’s important to acknowledge its limitations. Firstly, the tool relies on simulated patient interactions, which may not fully capture the complexities and nuances of real-life patient encounters. Although efforts have been made to create diverse scenarios, there might still be instances where the AI-generated responses don’t accurately reflect the range of patient experiences. Secondly, the tool’s effectiveness depends on the accuracy of the genetic and medical information generated by the AI. While measures have been taken to ensure plausibility and medical accuracy, there could still be instances where the AI produces inaccurate or unrealistic results.
It’s crucial for educators and students to be aware of these limitations and supplement the tool’s use with real-world experiences and guidance from qualified healthcare professionals. By recognizing and addressing these limitations, we can maximize the potential of the patient counseling tool while maintaining a cautious and critical approach to its application.
Conclusions and Future Implications:
The development of the patient counseling tool powered by OpenAI’s GPT-4 model represents a significant advancement in pharmacy education. By simulating patient interactions, providing personalized feedback, and facilitating skill development in pharmacogenetics, the tool equips students with the necessary competencies to excel in personalized medicine and individualized patient care. The tool’s ability to generate diverse scenarios and simulate realistic patient encounters allows for unlimited opportunities for skill development and learning.
Our tool is a first of its kind in Pharmacy education and offers an innovative solution to bridge the gap between theory and practice in pharmacy education. By simulating patient interactions, providing personalized feedback, and facilitating skill development in pharmacogenetics, the tool equips students with the necessary competencies to excel in personalized medicine and individualized patient care. As technology continues to advance and the field of pharmacogenetics evolves, the tool holds immense potential to shape the future of pharmacy education as well as those from other healthcare disciplines and contribute to improved patient outcomes.
Looking ahead, the patient counseling tool has several potential implications for the future of pharmacy education and practice. Firstly, as personalized medicine continues to advance, pharmacists will play a crucial role in interpreting genetic test results and tailoring treatments accordingly. The tool can be expanded to cover additional enzymes, genes, and drug interactions, providing students with a broader understanding of pharmacogenetics and further preparing them for this evolving aspect of healthcare.
Moreover, advancements in AI technology, such as enhanced language models and improved natural language processing, can contribute to the refinement and improvement of the patient counseling tool. By leveraging these advancements, the tool can better simulate realistic patient interactions, incorporate more diverse scenarios, and provide even more accurate and relevant information.
Additionally, the patient counseling tool can be integrated into pharmacy curricula and training programs, allowing students to access it at various stages of their education. By incorporating the tool early on, students can develop essential skills from the outset and build upon them throughout their training. This integration can also facilitate continuous learning and updates as new research and discoveries emerge in the field of pharmacogenetics.
Finally, in this article we outline the principles of designing a system message including the background, medical history, and rules for the AI simulated patient. This comprehensive approach allows for the customization of scenarios and the incorporation of various clinical situations, providing students with a wide range of learning experiences. This approach can be used to leverage AI to create new training scenarios for pharmacy students to enhance their education in areas besides pharmacogenetics.
Our tool offers an innovative solution to bridge the gap between theory and practice in pharmacy education. As technology continues to advance and the field of pharmacogenetics evolves, the tool holds immense potential to shape the future of pharmacy education and contribute to improved patient outcomes.
The above blog post, code and demonstration videos were created in partial fulfilment of the Digital Health Advanced Pharmacy Practice Elective rotation at Virginia Commonwealth University, School of Pharmacy.
Shared under CC4.0 License