Using The MONAI Framework To Segment Medical Images
Use the MONAILabel 3D Slicer extension to segment biomedical images.
This is the fourth part of a series of blog posts describing 3D Slicer biomedical image viewer and its applications, progressing from installation and basic use to using automated segmentation extensions. The first and second part of the series can be found here, and the third and fifth parts of the series discuss using the MONAI framework in 3D Slicer.
In part 3 of this series, we installed the monailabel Python package and the MONAILabel 3D Slicer extension onto our workstation, and learnt about the models available in the MONAILabel Radiology application for use in 3D Slicer. Now, let’s put it all to use.
First, we will use the Segmentation model to import an image volume and automatically segment it. Following that, we will do a quick tutorial of how you can train a model on a custom dataset to get to to the point where you can use the model for automatic segmentation.
Before we begin, we need to open Anaconda Prompt and activate our conda environment. Then we can run the models that we installed to this environment in the previous article.
Open Anaconda Prompt and run the following command:
conda activate MonaiNow our previously created Monai environment with all of our software installed is activated and ready to use. Let’s get started.
Using The Segmentation model to automatically segment structures
The first model that we will use is a pretrained segmentation model that is trained to segment 22 structures. We will download and import a dataset and use the Segmentation model to segment the 22 pretrained organs automatically, without any user input.
You will notice that this model is similar to the Total Segmentator extension — in fact, it is trained on the Total Segmentator dataset. Further, there is a model available in the Monai Model Zoo (more about that later) that will segment the same 104 structures as the TotalSegmentator extension that we used in part 3 of this series. See the 104 organs that can be segmented.
Before we begin we will download a sample data set. Head to this page and download “Chaos_Test_sets.zip”. You will have to create an account and then click the file name on this page and the download will begin. Unzip the file to your C:/Users/Username/datasets folder that we created in the previous article.
Let’s explore the structure of the files that we just downloaded. This dataset contains MR and CT DICOM volumes. Each volume has its own folder which contains separate files for every slice in the volume.
This is not the correct file structure that MONAILabel uses — MONAILabel wants each of those single slices (in the image on the right above) merged into a single file, and all of those new files placed in a single folder. With the correct file structure, you have the ability load your dataset in the command that you use to start the server, and to use the Next Sample button in the MONAILabel module to load volumes from your dataset into Slicer.
We could convert these DICOM images to NIFTI files with 1 file per volume, so that they are in the correct structure for MONAILabel. However, we are going to skip that step as it requires another Python package to be installed.
As a result, we will not provide the dataset location to the start_server command. Instead, we will load the whole DICOM folder into 3D slicer using the Slicer GUI (Graphic User Interface- the app that you communicate with your mouse and keyboard ). Then, we can load each volume in the DICOM module and switch back to the MONAILabel module.
Open 3D Slicer and in the top left corner select File->Add DICOM data. Navigate to the folder that you downloaded the dataset into and add the entire folder. This will import all of the subfolders (each one being a seperate image volume) into 3D Slicer.
Alternatively, you could drag and drop each folder into 3D Slicer. Both methods will result in any DICOM volumes in the added folders being available in the Add DICOM Data module:
With the images loaded, we are going to launch MONAILabel with the segmentation model running. Then, we will individually load each DICOM volume into MONAILabel.
Since this is an automatic segmentation model, once the images are imported, all the user has to do to automatically segment the structures is to hit Run. This functionality, of importing a dataset and immediately hitting Run, would be the end result of the below process of training a model.
Let’s start a server with the segmentation label. We will pass a location to the — -studies command as the command expects a path and will not execute correctly if we do not supply one.
monailabel start_server --app apps/radiology --studies datasets --conf models segmentation
In the ADD DICOM module, select a volume and hit Load. Note the study ID that I have selected is “9”, and the series description is “3”.
Navigate back to the MONAILabel module. Click the green arrow to connect to the server. Now, navigate to the Segment Editor where you will see the labels of the structures that are available to be segmented.
Now, under the Auto Segmentation tab, simply click Run. The segmentation will complete automatically.
If your 3D Slicer program is not responding, this is due to the segmentation running. Do not worry; the program will respond once the segmentation is complete. You can continue any other work on your workstation. It took 5–10 minutes to complete the segmentation on my CPU, and will be much quicker if you have a GPU.
Once the segmentation completes, head to the Segment Editor and click Show 3D. You may need to center the segmentation in the 3D viewer.
Saving Segmentations and labels
Now let’s save the segmentations. For further information you can read the Slicer docs on segmentation.
In the below screenshot, we only need to save the segmentation series. We can save it in in NRRD (.nrrd or .nhdr) and Nifti (.nii or .nii.gz) file formats by hitting the SAVE button highlighted in red.
Alternatively, you can save the segmentations in the Segmentations module. Under Export to File you will see the available options. Choose a format and directory and hit Export.
When saved, the segmentations can be found in the appropriate folder to be loaded into Slicer or whatever other software you have.
Success! That volume has been segmented with the labels available in the pre-trained Segmentation model, and the segmentations have been saved for quick loading later.
Next, let’s use the vertebrae segmentation pipeline model to quickly segment each vertebrae. Similar to the Segmentation model, we will simply start the monailabel with the vertebrae segmentation model and immediately hit Run to obtain our segmentation.
Using the automatic vertebrae segmentation model
To stop the previous model from running, in Anaconda Prompt hit Ctrl + C. Then confirm with Y. You should be able to see the server stop running in Anaconda Prompt.
Now we will restart the server with the vertebrae models by typing the following command:
monailabel start_server --app apps/radiology --studies dataset --conf models localization_spine,localization_vertebra,segmentation_vertebraAgain, we did not use the “ — studies” command correctly. We can not use the Next Sample button in MONAILabel and instead must manually load our data as before.
Close the scene in 3D Slicer to start fresh. Head to the DICOM module and select a study to segment, or download one of the 3D Slicer sample abdomen datasets. Return to the MONAILabel module and reconnect using the Green refresh arrow. You will now see the vertebrae model in your MONAILabel module:
Click Run. Allow the model to run and you will see the result. Each vertebrae is individually segmented.
Similar to the Segmentation model, you can see how the vertebrae model quickly segments structures that it was trained on.
But how does a model get to this point? In the next section, we will cover the steps to import a dataset and train your own model using the MONAILabel DeepEdit model. Once a model is trained on enough data, it can be saved and used at any time in 3D Slicer simply by loading the model and a dataset and hitting Run.
Using the DeepEdit model and Scribbles to segment structures
Now that you have seen how a model can automatically segment images in MONAILabel, let’s walk through the process of training a model to be able to do so. The outcome of this section is that you will have a pre-trained model, like above, that you can load into the start_server command and begin to automatically segment labels that you desire.
We will go through an example using the DeepEdit model and the new Scribbles tool that comes with MONAILabel on a sample dataset. The DeepEdit model is pretrained with 7 segments from the BTCV dataset. However, like all the models, it can be trained with new labels. We are going to go through this process.
Again, to stop the previous model from running, in Anaconda Prompt hit Ctrl + C. Then confirm with Y. You should be able to see the server stop running in Anaconda Prompt.
Let’s get some data. MONAI has provided the Medical Decathalon datasets for us to experiment with. Let’s see what is available.
monailabel datasets
Let’s download the sample Spleen dataset into a location “datasets”:
# download Task 2 MSD dataset
monailabel datasets --download --name Task09_Spleen --output datasetsNow, start the server and define which radiology app we want to use. In this case, we are going to use the DeepEdit model.
We will pass the use_pretrained_model false method with the command. This method turns off the pretrained DeepEdit model and allows you to train a model from scratch using the underlying Convolutional Neural Network (which we will do in the next section).
Type:
# start the Radiology app in MONAI label server
# and start annotating the downloaded images using deepedit model
monailabel start_server --app apps/radiology --studies datasets/Task09_Spleen/imagesTr --conf models deepedit --conf use_pretrained_model falseLet’s see the file structure of the folder we provided. For this example we do not have any labels so we just have to place our images in the dataset folder that we reference when we start the server. Contrary to above, this file structure will allow use to use the Next Sample button in the MONAILabel Slicer module to load volumes to segment.
The command also downloaded and started a pretrained DeepEdit model using a dynamic Unet. You will now see the server information in the console. You can monitor the activity in the terminal. You will need to leave this server running in the background, if you exit you will lose access to the models in 3D Slicer.
Okay! We have our server running, our DeepEdit model ready and the Spleen dataset ready to work on. Let’s use this application and quickly segment the spleen.
Press the green refresh button to connect to the server.
We will be using the untrained DeepEdit model plus the Scribbles tool to segment the spleen. You will see the selection of models you started the server with in a drop down list.
We will only see DeepEdit as it is the only model that we started the server with. Since we fed our dataset location to the — studies command we are able to use the Next Sample button to load out data, rather than loading volumes manually. Next Sample will load the next image from our dataset, choosing randomly which volume to load.
Hit Next Sample and an image from the dataset will load.
Now we will use the Scribbles to quickly segment an organ. You can use any segmenting tool you want, such as Grow From Seeds, scissor, paint, brush, but we will demonstrate the new Scribbles tool.
To begin, select a label. We will select spleen. First you have to place an ROI. Drop it on the spleen and use all 3 orthogonal views and ensure it is completely covered by the ROI.
Now select Foreground and hit the Paint button. Using the brush, paint the foreground and background of within the ROI. You just need a mark in each view for both the structure and the background. Hit the Update button. Hitting the Update button will run inference using the Scribbles you grew with the DeepEdit model and create a 3D volume.
Navigate to the segment editor and hit Show 3D. You will probably want to center the volume in the 3D viewer.
You will now see the segmentation. Modify the segmentation as we covered in earlier videos using Segment Editor tools or with Scribbles to refine the segmentation and hit Update again. The DeepEdit model will run again and the 3D volume will update. Repeat this process until you are satisfied with the segmentation.
You may want to also save your segmentation as you normally would. However, for our work with MONAILabel, when you are satisfied with the segmentation, hit Submit Label. This will save the segmentation label you created. As we mentioned before about the file structure, we can see this label was saved in the labels/final folder.
There! You segmented the spleen fairly easily. If you go the labels folder, you can see you now have one label created. You can do whatever you like with this segmentation. We discussed ways to extract save segmentations in the previous article, if you have uses for them. If not, we can continue to segment more images or train the model.
With the spleen completed, you can choose to segment other structures that the pretrained Deep Edit model is trained to segment. Select each organ from the Label dropdown list and repeat the process, placing and ROI and painting Scribbles over each organ that you would like to segment. Repeat the process and you will find all of the labels you created in the labels/final folder.
You can now choose to segment more organs or hit Next sample and repeat the process with a new CT volume. You can segment using Grow From Seeds or any Segment Editor tool that you like and continue the process. The MONAILabel extensions provides the inference from the labels that you create and it does this in the background with the apps we are running.
However, that is just the beginning of the MONIALabel training process. We will quickly touch on how we can use this label we just created to further train a more accurate model, ultimately having a model that will allow us to automatically create segmentations by importing data and hitting Run under auto segmentation.
Training A Model
See here and here for video demonstrations.
With your image submitted, you can begin to train the DeepEdit model. To begin training a model, click the Train button.
Note — This is a just an example of the workflow in 3D Slicer. There are changes that would need to be made in the configuration files in order to successfully train a new model from scratch. While we are just highlighting the method for example, further instructions can be found here if one is interested.
The training process will start in the background. You can continue to segment more images while it is running, adding them into the model training set as you segment them. You can view the training process as it executes in Anaconda Prompt (you will not understand anything, but it is interesting to see).
Once enough labels are segmented and provided and the model is trained for long enough, you will see the accuracy metrics improve.
Once the accuracy is 75–80%, the model is able to be saved. At this point, you can start using the model to segment new images without training, saving you significant time.
When the training completes, and validation data are correctly set. The latest trained model is automatically saved to “radiology/models” folder. You can find the model checkpoint there and use/call it at later time².
To open and run the model, it is just like before. Run the start_server command with the location of the data you want to segment and the name of the saved model (you will find the name of the model in radiology/models).
monailabel start_server --app apps/radiology --studies path/dataset --models conf modelnameYou will no longer have to provide any input with segmentation tools. As we did above, you may simply put new images into your train folder, start a server, hit Next Sample, and the Run button will now be available. Hit the Run button and run the model to get a segmentation.
There! That is all it takes to be able to automatically segment any structure you want. It may seem like a lot — so please see the following example of how this method can be completed.
Removing Segmentation Labels
To remove a segment class completely: Select the segment you would like to delete then click Remove segment to delete from the segmentation². To remove a predicted mask but not the segment label: Right click the segment and click clear selected segments. This will remove the mask and you can start the segmentation over from scratch.
Example — A Clinicians Workflow
I would like to highlight a workflow undertaken by Dr. Rudolf Bumm, a clinician familiar with the MONAILabel extension. In these two videos, Dr. Bumm recounts how he used LungCTSegmenter (mentioned in part 2 of this series) to automatically create lung segmentation labels. He then trained a model in MONAILabel using those created labels.
Dr Bumm’s method highlights the power of using the MONAI framework on it’s own or along with the other tools and extensions available in 3D Slicer to save significant time creating image segmentations. Using just 2 modules in 3D Slicer (LungCTSegmenter and MONAILabel) Dr. Bumm was able to create a powerful workflow that can save significant time. Any user that with MONAIALabel could use this workflow to import data and segment models with any of these tools and save a significant amount of time.
Monai Reviewer
If you have 20–30 minutes, this is an interesting video demonstrating the use of MONAILabel Reviewer for collaboration across teams of annotators.
Conclusion
In this article, we have discussed and used the MONAIALabel 3D Slicer extension. You should now understand how the extension is able to automatically segment images using trained models, as well as the process of how to get a model to that point.
Along the way we:
- Downloaded an open source data set
2. Automatically segmented the dataset using the Segmentation model
3. Automatically segmented the spine using the vertebrae_pipeline
4. Segmented structures using the interactive DeepEdit model
5. Used those segmented structures to train a model for automatic segmentation in the future
6. Introduced the MONAI Reviewer extension for collaboration across teams
As clinicians, you may not necessarily want to train models yourself. Luckily, there is a Model Zoo FULL of pre-trained models that areready to automatically segment your images! We will explore some of these models in the final article of this series.
Up Next: Custom models from the Monai Model Zoo
In part 5 of this series we will continue our usage of the MONAILabel extension to segment images. We will download custom models from the Monai Model Zoo and use them to segment and localize structures of the body, brain, and lungs.
References
[1] Project-MONAI. (n.d.). Project-Monai/Monailabel: Monai label is an intelligent open source image labeling and learning tool. GitHub. Retrieved April 28, 2023, from https://github.com/Project-MONAI/MONAILabel
[2]Segmentations¶. Segmentations — 3D Slicer documentation. (n.d.). Retrieved April 28, 2023, from https://slicer.readthedocs.io/en/latest/user_guide/modules/segmentations.html
