Evaluating lensless imaging reconstructions
In a previous tutorial, we developed a few reconstruction approaches for lensless imaging. Now we would like to evaluate the performance of our algorithms. To this end, we will use some common metrics in image processing and compute average metrics over a dataset.
This tutorial assumes that you have followed this one, and have downloaded and installed the LenslessPiCam repository.
Note that we have denoted exercises for students as such:
This is an exercise.
If you are an instructor / trying to replicate this tutorial, feel free to send an email to eric[dot]bezzam[at]epfl[dot]ch.
1) Common metrics
Below are some common metrics for determining the similarity between two images.
- Mean squared error (MSE): lower is better with a minimum of 0.
- Peak signal-to-noise ratio (PSNR): higher is better with values given in decibels (dB).
- Structural similarity index measure (SSIM): higher is better with a maximum of 1.
SSIM is specifically developed for images, as it exploits the idea that pixels have strong inter-dependencies, especially when they are spatially close together. Consequently, SSIM does a better job at taking textures into account.
With the rise of deep learning, a new breed of metrics has been proposed, which use the deep features of trained neural networks (i.e. passing the images through the model) in order to determine the similarity between images. A very popular one is known Learned Perceptual Image Patch Similarity (LPIPS), which works on patches. Lower is better with a minimum of 0.
Python implementations of the above metrics can be easily found online:
- MSE: https://scikit-image.org/docs/stable/api/skimage.metrics.html#skimage.metrics.mean_squared_error
- PSNR: https://scikit-image.org/docs/stable/api/skimage.metrics.html#skimage.metrics.peak_signal_noise_ratio
- SSIM: https://scikit-image.org/docs/stable/api/skimage.metrics.html#skimage.metrics.structural_similarity
- LPIPS: https://github.com/richzhang/PerceptualSimilarity
We have provided a wrapper to these functions within the LenslessPiCam library.
Within the context of lensless imaging, we would like to compute these metrics between the ground-truth image and our reconstruction. If we do not have the ground-truth image, we cannot use these metrics. However, we could still use the data fidelity term, as we saw in our optimization, in order to judge the quality of our reconstruction.
where y is our raw measurement, H is our forward model, and x’ is our reconstruction. While this metric has the advantage of not requiring the ground-truth image, there are a couple problems with using it:
- As y is not the target image, we may miss out on “penalizing” some features from the original image that we would like to capture.
- The forward operator H may very different from practice, e.g. wrong / too ideal assumptions.
2) Comparing with original image
Let’s see how we can apply the above metrics to our lensless imaging reconstruction. This script from the LenslessPiCam library takes as input the file paths to the reconstruction and the original image (defined in a configuration file):
python scripts/compute_metrics_from_original.py \
alignment.vertical_crop=null \
alignment.horizontal_crop=null \
alignment.rotation=0Note that the reconstruction file is an NPY file. You can see here how that can be saved.
Applying the script will print out some metrics:
MSE 0.762889523472075
PSNR 1.1753834909664385
SSIM 0.022794632
LPIPS 0.6300908923149109However, the visual output may show that we are comparing very different regions.
That is because we need to specify the region in the reconstruction over which we would like to compute the metric: vertical_crop, horizontal_crop, and rotation.
python scripts/compute_metrics_from_original.py \
alignment.vertical_crop=[262,371] \
alignment.horizontal_crop=[438,527] \
alignment.rotation=-0.5Below are the printed metrics and corresponding visual output.
MSE 0.16410427100550945
PSNR 7.8488011577818595
SSIM 0.40505564
LPIPS 0.6445358991622925A quick comparison of the metric value (before and after region selection) show some pitfalls for particular metrics. Notice how MSE gets worse after region selection!
Determining the appropriate values for vertical_crop, horizontal_crop, and rotation will require some trial-and-error. In order to avoid doing this for a large set of files, we recommend displaying images with the same dimensions and keeping your LenslessPiCam setup fixed. That way, the same vertical_crop, horizontal_crop, and rotation values can be used for all of your reconstructions.
Exercise: determine the
vertical_crop,horizontal_crop, androtationvalues for your reconstruction(s).
3) Comparing with data acquired from a parallel setup
It may be unreasonable to expect the LenlessPiCam reconstruction to be able to recover the original image due to limitations of the display device, optical elements, and/or the camera sensor. In fact, a lensed measurement may exhibit similar artifacts. For a more fair comparison, one can utilize a parallel setup as proposed in this paper and shown below.
In the above setup, a screen is used to display the original image. This image is diverted into two path through the use of a beam splitter, such that we capture the original image with a classical lensed camera and LenslessPiCam (lensless camera). Our ground-truth for metrics now becomes the measurement made by the lensed camera. A crucial aspect of the above setup is a pixel-wise alignment between the two cameras. This is described in Section 5 of the paper.
DiffuserCam Lensless Mirflickr Dataset
Setting up the above acquisition system is not a trivial task. Fortunately, the authors of the above paper have acquired and released a dataset of 25'000 image pairs known as the DiffuserCam Lensless Mirflickr dataset (DLMD). It is available for download on their website.
The original dataset is 100 GB unzipped. So in case that is too large for your computer, we’ve prepared a subset of 200 image pairs (725 MB) with viewable lensless/lensed data.
This script from the LenslessPiCam repository can be used to evaluate ADMM on 10 files and save the reconstructions (default parameters defined in this configuration file).
python scripts/evaluate_mirflickr_admm.py \
n_files=10 save=TrueThe script will print out metrics for each file and average metrics.
...0.1895594637860311
7.222545284759978
0.39497176
0.6669930219650269Reconstructions saved to : XXX/LenslessPiCam/scripts/admm_mirflickr_DDMMYEAR_HHhMMMSE (avg) 0.11076617020676796
PSNR (avg) 10.766469050811647
SSIM (avg) 0.49509287
LPIPS (avg) 0.6319254070520401
Below are some of the reconstruction, with the corresponding raw (lensless) and lensed data.
To focus on a single image and visualize its reconstruction as the number of iterations increase, the following script can be run. Default parameters defined in this configuration file.
python scripts/apply_admm_single_mirflickr.pyBelow is an example output.
Exercise: apply your own reconstruction to the DiffuserCam MIR Flickr dataset (or a subset of it) in order to compare their performances with that of ADMM, as in Figure 4 and Table 2 of the paper “Learned reconstructions for practical mask-based lensless imaging”: https://arxiv.org/pdf/1908.11502.pdf
