How I Conducted My Research
The methods and results sections from my MPH capstone (link here) for UC Berkeley’s School of Public Health.
Visit my Building Trust and Adoption in Machine Learning in Healthcare site (link here) for the abridge and full version of my MPH capstone as well as upcoming summaries of interviews and additional research.
Primary and Secondary Research Methods
Clinicians interviews
I sought out clinicians with the intention of gathering diverse viewpoints. Inclusion criteria can be found in Figure 1 below, with some case-by-case exceptions.
Interviews occurred during a seven-week period, starting early February 2020, and were based on a consistent interview guide (link here). I scoped interviews to 30 minutes by phone and they often ran longer. I also both led the interviews and took notes.
My goal was to further the knowledge base of clinicians’ perceptions of ML in healthcare in an open source philosophy. Therefore after each interview, notes were summarized and published, pending approval from each interviewee (link here). Name, location, gender, employer, and other identifying information were removed and language was standardized for ease of reading and consistency.
For drafting the capstone (link here), I collated all raw and un-summarized notes into one document, allowing me to review all responses by interview question. Key themes and insights were gathered through an iterative approach with supporting quotations appended to each insight.
Secondary research
I conducted secondary research to both provide a background on trust and adoption of ML in healthcare and to validate individual perspectives of clinicians.
I sourced journal articles from the most influential medical journals according to Wikipedia reference frequency, the Doctor Penguin weekly newsletter, and leading Twitter accounts (link here). Given the fast pace of the space, priority was given for articles published January 1, 2019 through Spring 2020.
Articles from popular media were sourced from a regular review of email newsletters that I subscribe to (link here). A limited number of these articles were directly sourced in this paper, yet many more provided an underlying understanding of the space.
Zotero was used for citation management.
Research Results
By the end of seven weeks, I was able to interview 18 clinicians. This was just before the COVID-19 pandemic officially shut down parts of the country. Clinicians had diverse backgrounds and could speak to a range of topics found in Figure 2 below.
Key themes naturally arose based on the structure of my interview guide, and the Discussion section is organized as such in Figure 3 below.
References
Below is the list of the many references that informed my research, including the interviews, journal articles, and popular media articles.
Abid, A., Abdalla, A., Abid, A., Khan, D., Alfozan, A., & Zou, J. (2020). An online platform for interactive feedback in biomedical machine learning. Nature Machine Intelligence, 2(2), 86–88. https://doi.org/10.1038/s42256-020-0147-8
Allergist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
American College of Radiology Data Science Institute. (n.d.). Define-AI Directory. Retrieved April 13, 2020, from https://www.acrdsi.org/DSI-Services/Define-AI
Blease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019). Artificial Intelligence and the Future of Primary Care: Exploratory Qualitative Study of UK General Practitioners’ Views. Journal of Medical Internet Research, 21(3), e12802. https://doi.org/10.2196/12802
Cardiologist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
CB Insights Research. (2020). AI In Numbers Q1’20: Global Funding, Corporate Activity, Partnerships, And R&D Trends. https://www.cbinsights.com/research/report/ai-in-numbers-q1-2020/
Chen, E., Rajpurkar, P., Topol, E., & Ng, A. (n.d.). Doctor Penguin. Retrieved April 15, 2020, from http://doctorpenguin.com
Consumer Technology Association. (2020). Definitions/Characteristics of Artificial Intelligence in Health Care (ANSI/CTA-2089.1) (p. 32). Consumer Technology Association. https://shop.cta.tech/products/definitions-characteristics-of-ai-in-health-care
Craft, L. (2019a, February 25). Healthcare Provider CIOs: Get Ahead of AI Innovation With Strong AI Governance. Gartner. https://www.gartner.com/document/3902977
Craft, L. (2019b, August 2). Understand the Value of AI for Healthcare Delivery Organizations. Gartner. https://www.gartner.com/document/3869974
Craft, L., & Jones, M. (2019). Hype Cycle for Healthcare Providers, 2019. Gartner. https://www.gartner.com/document/3953717
Craft, L., & Singh, P. (2020). State of AI — Healthcare Providers’ Perspective. Gartner. https://www.gartner.com/document/3979188
Cruz, L. P., & Treisman, D. (2020). Making AI Great Again: Keeping the AI Spring. 144–151. http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006896001440151
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., & Mandl, K. D. (2020). Machine intelligence in healthcare — Perspectives on trustworthiness, explainability, usability, and transparency. Npj Digital Medicine, 3(1), 1–5. https://doi.org/10.1038/s41746-020-0254-2
Dermatologist. (2020, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Emanuel, E. J., & Wachter, R. M. (2019). Artificial Intelligence in Health Care: Will the Value Match the Hype? JAMA, 321(23), 2281–2282. https://doi.org/10.1001/jama.2019.4914
Emergency Medicine Physician. (2020, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Emergency Medicine Physician. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Endocrinologist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Faes, L., Wagner, S. K., Fu, D. J., Liu, X., Korot, E., Ledsam, J. R., Back, T., Chopra, R., Pontikos, N., Kern, C., Moraes, G., Schmid, M. K., Sim, D., Balaskas, K., Bachmann, L. M., Denniston, A. K., & Keane, P. A. (2019). Automated deep learning design for medical image classification by health-care professionals with no coding experience: A feasibility study. The Lancet Digital Health, 1(5), e232–e242. https://doi.org/10.1016/S2589-7500(19)30108-6
Fernandez Garcia, J., Spatharou, A., Hieronimus, S., Beck, J.-P., & Jenkins, J. (2020). Transforming healthcare with AI: The impact on the healthcare workforce and organisations (p. 134). EIT Health, McKinsey & Company. https://eithealth.eu/our-impact/our-reports/report-transforming-healthcare-with-ai/
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019). Adversarial attacks on medical machine learning. Science, 363(6433), 1287–1289. https://doi.org/10.1126/science.aaw4399
Gastroenterologist. (2020, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Geis, J. R., Brady, A. P., Wu, C. C., Spencer, J., Ranschaert, E., Jaremko, J. L., Langer, S. G., Borondy Kitts, A., Birch, J., Shields, W. F., van den Hoven van Genderen, R., Kotter, E., Wawira Gichoya, J., Cook, T. S., Morgan, M. B., Tang, A., Safdar, N. M., & Kohli, M. (2019). Ethics of Artificial Intelligence in Radiology: Summary of the Joint European and North American Multisociety Statement. Radiology, 293(2), 436–440. https://doi.org/10.1148/radiol.2019191586
Gerke, S., Babic, B., Evgeniou, T., & Cohen, I. G. (2020). The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. Npj Digital Medicine, 3(1), 1–4. https://doi.org/10.1038/s41746-020-0262-2
Gibson, W., & Brin, D. (2018, October 22). The Science in Science Fiction [NPR Podcast]. https://www.npr.org/2018/10/22/1067220/the-science-in-science-fiction
Goecks, J., Jalili, V., Heiser, L. M., & Gray, J. W. (2020). How Machine Learning Will Transform Biomedicine. Cell, 181(1), 92–101. https://doi.org/10.1016/j.cell.2020.03.022
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019). The practical implementation of artificial intelligence technologies in medicine. Nature Medicine, 25(1), 30–36. https://doi.org/10.1038/s41591-018-0307-0
Hospitalist. (2020, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Hospitalist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Hwang, T. J., Kesselheim, A. S., & Vokinger, K. N. (2019). Lifecycle Regulation of Artificial Intelligence– and Machine Learning–Based Software Devices in Medicine. JAMA, 322(23), 2285–2286. https://doi.org/10.1001/jama.2019.16842
Jemielniak, D., Masukume, G., & Wilamowski, M. (2019). The Most Influential Medical Journals According to Wikipedia: Quantitative Analysis. Journal of Medical Internet Research, 21(1), e11429. https://doi.org/10.2196/11429
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17(1), 195. https://doi.org/10.1186/s12916-019-1426-2
Kennedy, G., & Gallego, B. (2019). Clinical prediction rules: A systematic review of healthcare provider opinions and preferences. International Journal of Medical Informatics, 123, 1–10. https://doi.org/10.1016/j.ijmedinf.2018.12.003
Larson, D. B., Magnus, D. C., Lungren, M. P., Shah, N. H., & Langlotz, C. P. (2020). Ethics of Using and Sharing Clinical Imaging Data for Artificial Intelligence: A Proposed Framework. Radiology, 192536. https://doi.org/10.1148/radiol.2020192536
Littmann, M., Selig, K., Cohen-Lavi, L., Frank, Y., Hönigschmid, P., Kataka, E., Mösch, A., Qian, K., Ron, A., Schmid, S., Sorbie, A., Szlak, L., Dagan-Wiener, A., Ben-Tal, N., Niv, M. Y., Razansky, D., Schuller, B. W., Ankerst, D., Hertz, T., & Rost, B. (2020). Validity of machine learning in biology and medicine increased through collaborations across fields of expertise. Nature Machine Intelligence, 2(1), 18–24. https://doi.org/10.1038/s42256-019-0139-8
Liu, X., Rivera, S. C., Faes, L., Ferrante di Ruffano, L., Yau, C., Keane, P. A., Ashrafian, H., Darzi, A., Vollmer, S. J., Deeks, J., Bachmann, L., Holmes, C., Chan, A. W., Moher, D., Calvert, M. J., Denniston, A. K., & The CONSORT-AI and SPIRIT-AI Steering Group. (2019). Reporting guidelines for clinical trials evaluating artificial intelligence interventions are needed. Nature Medicine, 25(10), 1467–1468. https://doi.org/10.1038/s41591-019-0603-3
Marx, V. (2019). Machine learning, practically speaking. Nature Methods, 16(6), 463–467. https://doi.org/10.1038/s41592-019-0432-9
Matheny, M. E., Whicher, D., & Israni, S. T. (2020). Artificial Intelligence in Health Care: A Report From the National Academy of Medicine. JAMA, 323(6), 509–510. https://doi.org/10.1001/jama.2019.21579
Medical Registered Nurse. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Morley, J., & Floridi, L. (2020). An ethically mindful approach to AI for health care. The Lancet, 395(10220), 254–255. https://doi.org/10.1016/S0140-6736(19)32975-7
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020). Artificial intelligence versus clinicians: Systematic review of design, reporting standards, and claims of deep learning studies. BMJ, 368. https://doi.org/10.1136/bmj.m689
Nelson, C. A., Pérez-Chada, L. M., Creadore, A., Li, S. J., Lo, K., Manjaly, P., Pournamdari, A. B., Tkachenko, E., Barbieri, J. S., Ko, J. M., Menon, A. V., Hartman, R. I., & Mostaghimi, A. (2020). Patient Perspectives on the Use of Artificial Intelligence for Skin Cancer Screening: A Qualitative Study. JAMA Dermatology. https://doi.org/10.1001/jamadermatol.2019.5014
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019). Addressing Bias in Artificial Intelligence in Health Care. JAMA, 322(24), 2377–2378. https://doi.org/10.1001/jama.2019.18058
Park, S. H., Do, K.-H., Kim, S., Park, J. H., & Lim, Y.-S. (2019). What should medical students know about artificial intelligence in medicine? Journal of Educational Evaluation for Health Professions, 16. https://doi.org/10.3352/jeehp.2019.16.18
Pinto dos Santos, D., Giese, D., Brodehl, S., Chon, S. H., Staab, W., Kleinert, R., Maintz, D., & Baeßler, B. (2019). Medical students’ attitude towards artificial intelligence: A multicentre survey. European Radiology, 29(4), 1640–1646. https://doi.org/10.1007/s00330-018-5601-1
Plastic & Reconstructive Surgeon. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Polesie, S., Gillstedt, M., Kittler, H., Lallas, A., Tschandl, P., Zalaudek, I., & Paoli, J. (n.d.). Attitudes towards artificial intelligence within dermatology: An international online survey. British Journal of Dermatology, n/a(n/a). https://doi.org/10.1111/bjd.18875
Price, W. N., Gerke, S., & Cohen, I. G. (2019). Potential Liability for Physicians Using Artificial Intelligence. JAMA, 322(18), 1765–1766. https://doi.org/10.1001/jama.2019.15064
Primary Care Physician. (2020a, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Primary Care Physician. (2020b, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Psychiatrist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Radiation Oncologist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Radiologist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Rajkomar, A., Dean, J., & Kohane, I. (2019). Machine Learning in Medicine. New England Journal of Medicine, 380(14), 1347–1358. https://doi.org/10.1056/NEJMra1814259
Rampton, V., Mittelman, M., & Goldhahn, J. (2020). Implications of artificial intelligence for medical education. The Lancet Digital Health, 2(3), e111–e112. https://doi.org/10.1016/S2589-7500(20)30023-6
Sarwar, S., Dent, A., Faust, K., Richer, M., Djuric, U., Van Ommeren, R., & Diamandis, P. (2019). Physician perspectives on integration of artificial intelligence into diagnostic pathology. Npj Digital Medicine, 2(1), 1–7. https://doi.org/10.1038/s41746-019-0106-0
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020). Presenting machine learning model information to clinical end users with model facts labels. Npj Digital Medicine, 3(1), 1–4. https://doi.org/10.1038/s41746-020-0253-3
The Medical Futurist. (2020). FDA-approved A.I.-based algorithms. The Medical Futurist. https://medicalfuturist.com/fda-approved-ai-based-algorithms
Trauma Registered Nurse. (2020, February). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Urologist. (2020, March). Clinician Interview (H. Goldberg, Interviewer) [Personal communication].
Vollmer, S., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K. S. L., Myles, P., Grainger, D., Birse, M., Branson, R., Moons, K. G. M., Collins, G. S., Ioannidis, J. P. A., Holmes, C., & Hemingway, H. (2020). Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ, 368. https://doi.org/10.1136/bmj.l6927
Wang, F., Kaushal, R., & Khullar, D. (2020). Should Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine? Annals of Internal Medicine, 172(1), 59. https://doi.org/10.7326/M19-2548
What is AI? (2014, September 5). The Society for the Study of Artificial Intelligence and Simulation of Behaviour. https://aisb.org.uk/what-is-ai/
