How computers are saving mice (and maybe humans)
An Augmented Reality setup to improve accuracy in the 5/6 nephrectomy surgical model
In the past 10 years or so, humanity has advanced at a pace hitherto unknown and has surpassed any expectations we may have set for ourselves. The driving force behind this boom in technological development can be attributed to the meteoric rise of increasingly efficient and large scale computation systems. This is very apparent when looking at the rise of autonomous vehicles but even more impressive is the ability to bring this technology to smartphone, not dissimilar to the approach comma ai has taken. If you had told any researcher or engineer that one day, we would be able to have our cars drive themselves (albeit semi-autonomously), let alone on a smartphone, this would have been unbelievable. One field that has benefited enormously from these advancements has undoubtedly been medicine. In all facets of clinical practice, the digital age has transformed how we think about healthcare. For example we have systems that allow us to perform surgery across thousands of kilometres with remote assistant systems like DaVinci and powerful algorithms that can analyze a patient's genetic data to provide meaningful insights into their treatment pathways. While these systems cost millions of dollars to develop and implement, there are still ways in which technology is helping in a clinical setting on a smaller scale. The big idea here is that technology enables us to solve problems at a blistering speed.
One of these issue is something known as chronic kidney disease (aka CKD), now you may know this as just kidney disease, or kidney failure, but it’s main characteristic is the loss of a kidneys ability to filter out impurities in our blood, over some time.
Now, this may seem like a problem that doesn’t affect a great deal of people, or perhaps affects those far away from home, but in the US alone, more than 35 million adults suffer from chronic kidney disease. This is a staggering number, but even more shockingly, there is currently no cure for kidney disease. So as clinical researchers, it’s quite important to understand this disease and find ways of simulating this loss of filtration in order to develop effective treatment pathways.
This brings us to the 5/6 nephrectomy, which is one of the above mentioned methods of simulating CKD in a more feasible model, in this case, mice. It starts with the removal of around 2/3 of the left kidney, followed one week later by the removal of the entire right kidney.
Now the way that these cuts are made is fairly haphazard and is completely reliant upon the skill and experience of the operators. As we can see here however, even operators with dozens of completed surgeries still struggle with consistently achieving the desired ratio. This on it’s own would not be such a large issue were it not for the repercussions of inconsistency in live mice models. If any less than 50% of the kidney is removed in the first stage, then it becomes impossible to draw any meaningful conclusions from the procedure and the mouse will be rejected from the study. should the amount be greater than 70% then the mouse is very likely to die during the surgery. In both cases the mice are rejected from the study wasting valuable time, resources and animal lives.
So the question really is, how can we decrease the variability in the amount of kidney that the operators remove with every procedure. One solution that we propose is to leverage computational techniques in order to inform the surgeon on exactly where to make these incisions thereby removing any guesswork while still allowing for human input. The three major systems we have chosen to tackle this problem with are Computer Vision (CV), Mathematical Modelling and Augmented Reality (AR) to enable us to deliver a user friendly method to the operator of exactly where to cut.
To begin with our solution we will go over the necessary hardware that we have in our current testbench prototype and what we intend to implement in our first iterations in a surgical setting. This includes a laptop with an attached webcam as well as an enclosure that helps us to mitigate occlusions and shadows. The system is meant to be self contained and mobile in order to test in various surgical locations and on our testbench. In addition we aim to have a platform that is not limited by computing power. As we can see the system is able to run with any integrated graphics on CPUs that are several years old.
This brings us to where we are today, with our current working application. What we can see is the algorithm, tracking the kidney throughout the frame and displaying various metrics, which are overlayed on the live camera feed. In addition, we can see that the algorithm is able to track the kidney not only through any x,y,z position in frame, but additionally through 360 degrees of rotation which is important for the analysis.
Now in Computer Vision, an important step in any project is always camera calibration. This may seem rather trivial, but it’s a very important step because of a single cameras inability to provide information about depth. take for example the three pictures below. They are all of the same model, with the same camera, lens and lighting, but we can see that they provide wildly different scale parameters at different heights to the camera. in order to solve this issue, we use something called an ARuco marker. this is a fiducial or reference object that helps us to normalize the objects in our scene. For example, we can see here how an aruco marker helps us to create a pixel to centimetre conversion of all objects in the scene and calculate an SI unit length for these objects. Now you may be asking, why an ARuco marker was chosen for this application, couldn’t any household object do the trick if we know it’s exact scale parametres? Well in short, yes. However, the reason for this markers popularity in the field is because it can also inform us about camera and lens distortions with further increase our precision on smaller lengths. It does this by means of the checkerboard/QR code pattern on the inside, and by measuring the characteristics of each of these squares we can quantify a distortion matrix for that specific camera. And keep in mind, these calibrations happen every single frame!
Once the camera is sufficiently calibrated, the next step is to find exactly where our kidney is in the frame. for a human, this is a fairly trivial task, but for a computer, the scene is a jumbled mess of organs, fur and surgical implements. For the purposes of our test bench, we use a simple background extraction algorithm to take advantage of the contrasted colour difference of our subject and the background. The algorithm uses a binary mask to pass over the image and return zeroes for anything that’s not our object and ones for everything else (i.e our kidney!). Now that we know where our kidney is, and we can track it throughout the frame, it becomes important to understand the size and orientation of our kidney. It becomes of particular importance later on, during the projection and cutpoint prediction stages. As for orientation, a standard method of bounding box generation tends to have the bounding box change parameters for each different orientation, changing it’s length and with to accommodate the object. While this approach may work well for many applications, in this case, we need the bounding box, in collaboration with the ARuco marker, to give us a constant size definition for our kidney. To solve this problem, we decided to make use of are minimization techniques to ensure that the bounding box rotates with the kidney, instead of changing shape to accommodate the rotations. This pose estimation also gives us the added benefit of PCA vectors which we will use in later steps.
Now, upto this point, we have been dealing with mainly 2 dimensional object recognition, modelling, tracking etc. It is important, however, to consider the real 3 dimensional aspect of our application before proceeding any further. On order to do this, we received 3D-MRI scans of an ex-vivo kidney to gain more information on the 3D aspect. These scans took place in an eppendorf tube, and were scanned in a formalin solution. we scanned multiple kidneys at a time to facilitate more data gathering.
Once again, we need to make a usable 3D model for our cutting pipeline and to do this from an MRI image, we needed to fo a couple of things. First is, splitting up the video into its distinct frames (for the uninitiated, MRI scans can be viewed as 2D videos through an axis) and then segmenting the kidney from the image into 3D slices. By recombining these slices, we are able to generate a usable 3D model. With this model, we can recreate our own 3D scan of the object, giving us a constant depth across every slice we obtain from our new model.
The slices now obtained, and a 3D model formed, the next step is accurately converting from a volumetric target, to something that can be easily illustrated (in real time) to the surgeon. We will start with calculating the volume of a single slice of our kidney. Now a kidney has many internal structure, and our MRI generated model had picked up on this, and what this meant was that each slice was an irregular shape with various cutouts in the middle. To solve this problem, we binarized the image of each slice, and using our standard area x depth = volume calculations (using the number of white pixels as our area and our constant depth as depth) to give us the volume for each slice independently. This gives us a means of calculating the volume of the kidney at any slice number of our choosing (by summation of all other slices below).
Why this is useful for us, as previously mentioned is for intuitive visualisation for the surgeon. With our standard 3D model, we can pre calculate our 5/6 ratio and split that over two segments to obtain our volumetric requirements. Now we can iterate over our kidney, from bottom to top and sum the slices until we reach our target volume as defined above! Once we know the number of slices, we can simply sum the depths of each slice to get a lengthwise measurement of how high up the kidney the cut must be! From there, we draw these cut points and overlay them on the image, using the correct scaling and orientation parameters we got previously.
Now, our current setup isn’t perfect, as discussed before, the current method of image segmentation (single pass binarization) would likely fail in the presence of a true surgical setting which is much more intricate and cluttered.
To solve this, we have built a Deep learning image segmentation model to identify and isolate our kidney in the image. This pipeline uses a custom semantic segmentation model to correctly identify the kidney in a live video stream.
Heres a quick summary of what we’ve achieved with this project
Our next iteration, in terms of our hardware will be a more well integrated setup, where the compute, I/O and visualization all take place in real time just above the operators hands. The improved efficiency will allow us to develop a smaller package and integrate better with our surgical environment.
Finally, I would like to thank all of my contributors for having supported this project from Mcmaster University and the Hamilton Centre for kidney research, based in St Joseph’s Hospital Hamilton.
For additional explanation, please see my presentation at the canadian association of physicists conference in 2022. https://youtu.be/I0ESYlp85Rk
If you would like to get more information or just want to discuss, I am always happy to chat, please feel free to send me an email at udbhavram41@gmail.com or visit my personal website.
