The Use of Augmented Reality in Surgery — Can you Imagine it?

Throughout November, I have been researching the use of Augmented Reality (AR) in surgery. The prospect of objects flying around in the operating room sounded like a practical joke that I would have laughed off had this not been a class assignment.

My first interaction with AR has been using the Live View feature in Google Maps and seeing the huge arrows showing me my next turn. Well, the same concept has been applied to the medical field, nowhere else than in surgery. Before you swear that you’ll never have your family member operated on, under those conditions, finish reading this article. I intend to share the knowledge gathered from my literature review on the topic, what AR is, how it is applied in surgery, and some breakthroughs they have made in this adventure.

Augmented Reality is a technology that places users in an interactive environment where computer-generated information is superimposed on the physical environment in real time. In simple terms, objects are generated on a computer; you see them through a lens (glasses or a screen) in real-time, but in actuality and with your natural eye, these objects are not present. The level of interactivity involves touch, haptic feedback (vibration), sound, etc. Sounds magical!

Short Story

Skull base endoscopy

Do you know the tissues and organs you have within your head, underneath your skull? Doctors in the past could access these tissues by simply ripping the face up (maybe ripping is an exaggeration, but they can open your skull or create holes in your face to access them). This, in the past, was causing too much damage to healthy tissues and prolonging healing time post-operation.

Through technologies like endoscopy, surgeons now get to these tissues through the nostril or a keyhole they will create. Endoscopy is a technology where doctors have a probe (tube/ wire-ish) with a camera attached to it; they manage to access surgical points that seem inaccessible and perform the procedures that need to be done. There is a separate screen (navigation, where he sees how he is controlling the direction of the camera) and a screen with pictures (scans) taken of the surgical target and the surgical field (patient in front of him). Combining all three views, he sees where he is going and performs the procedure.

endoscopy

This system works, only that there is too much mental load for the surgeon because he has to look at the pre-op scans (photos of the surgical site taken before), look at the 2D images the camera of the probe is capturing, imagine it in 3D, and execute the task.

Surgeon, during surgery

Medical technologists imagined, or are still imagining (heavily under research), a means to have both the navigation screen, the pre-op photos, and the operation field within the same field of view. This way, the surgeon is relieved of the mental load as well as facilitating the surgical process for them.

This is how AR comes in…

AR in surgery, according to Houssain et al., should provide supplemental anatomical as well as navigational information in 3D on a single display. Fundamentally, an AR system is supposed to see through structures and reveal hidden information without INTERFERING with the surgical process.

My interest lies in the interference, because how then do we implement such technology without it interfering with the surgery? I haven’t seen a paper that characterizes the interference.

How Does AR Work?

An AR system typically has cameras, display units, and trackers, and in the hospital, there would be goggles, head-mounted displays, joysticks, navigators, and co-manipulated robots among other hardware devices. These devices are connected based on these principles:

1. a 3D virtual environment is generated from a 2D image (be it a CT or MRI image). CT- computed tomography, a medical imaging technique to obtain images of the internal organs (think of x-ray images), MRI- Magnetic resonance imaging — generates images of internal organs as well;
2. the environment is registered (overlaid, superimposed) on the physical object (be it an organ or tissue);
3. a display unit that shows the combined view of the real and virtual world.

Benefits/Importance of AR in Surgery

1. AR is supposed to improve localization, navigation, and orientation in a surgical field by incorporating pre and intraoperative information. Pre-operative information could be information collected on the patient before the surgery, scans, history, etc. Intraoperative would be real-time imaging during the surgery, the location of the tumor, and changes happening in organs around the surgical target.

Localization determines the locus of the target during the surgery (the specific tumor or implant), navigation guides the surgical instruments to perform the task, and orientation is the position of the patient in relation to the surgical instruments.

2. Quick access to information on anatomical landmarks, patient history, and even research sites in real-time during surgery makes the entire process efficient and cost-effective. According to Jang W. Yoon et al., a hospital spends between $120 and $250 per minute during a surgical operation. Imagine some procedures taking 12 hours or even more per patient. With AR, the surgeon is freed of some repetitive tasks and solely concentrates on the task at hand, ergonomics is key in AR system developments.

3. Education and remote expert consultation. It has been proven that with AR, surgical residents, have access to quality practical materials in real-time and this enriches their training period. An American surgeon could assist in surgery in Asia, for example, through AR systems.

Could you think of problems that may arise due to the acceptance of AR into the normal surgical flow? Are there ethical issues that come up in the usage of AR systems? So far, I haven’t mentioned machine learning and artificial intelligence; do these technologies play a role in the success of AR systems?

In subsequent articles, I will talk about some principles that keep AR running, problems of AR in surgery, some popular AR tools that have been used in surgery, research papers that employed Deep learning techniques in AR, and how they dealt with the problems.

References

1. Hussain, R., Lalande, A., Guigou, C., & Bozorg-Grayeli, A. (2020). Contribution of Augmented Reality to Minimally Invasive Computer-Assisted Cranial Base Surgery. IEEE Journal of Biomedical and Health Informatics, 24(7), 2093–2106. https://doi.org/10.1109/JBHI.2019.2954003
2. Yoon, J. W., Chen, R. E., Kim, E. J., Akinduro, O. O., Kerezoudis, P., Han, P. K., Si, P., Freeman, W. D., Diaz, R. J., Komotar, R. J., Pirris, S. M., Brown, B. L., Bydon, M., Wang, M. Y., Wharen, R. E., & Quinones-Hinojosa, A. (2018). Augmented reality for the surgeon: Systematic review. In International Journal of Medical Robotics and Computer Assisted Surgery (Vol. 14, Issue 4). John Wiley and Sons Ltd. https://doi.org/10.1002/rcs.1914