Published in


Google’s AI can see through your eyes what doctors can’t

Can you tell from the photo below if the patient is male or female?

Google’s AI can tell gender by simply looking at a retinal photo like this one

Human ophthalmologists can only take a guess, and there is a 50:50 chance of getting it right or wrong.

Google’s AI, however, can predict it very well (AUC 0.97).

How is it possible? We don’t know yet but these results are definitely striking. Perhaps the neural network picked up patterns that are too subtle to the human eye.

To better understand this AI “black box”, a deep-learning technique called soft attention was used to identify the anatomical regions that are most important for the model to generate these gender predictions.

Attention maps (saliency or heat maps) were generated, representing what the model was“looking at” in each fundus photo in order to predict gender. These maps were then given to 3 ophthalmologists to identify the highlighted features in these maps. Ophthalmologists were blinded to the output predictions.

soft attention heat map is overlain in green, indicating the areas of the heat map that the neural-network model is using to make the gender prediction for the image.

100 heat maps were randomly selected and interpreted by ophthalmologists. They found that 71% of the heat maps highlighted vessels; 78% of the maps highlighted the optic disc, and 50% of the maps also highlighted non-specific features. In other words, the neural network was looking at a bit of everything but nothing in particular…

Until this day, we didn’t think that there were any differences between male and female eyes. How can a neural network tell them apart from just a photo of the retina is yet to be elucidated.

Although predicting gender from a retinal photo isn’t directly useful in patient care, this discovery will guide future research to investigate this gender difference, previously unknown to us. We can be hopeful that a better understanding of the anatomy and physiology differences between the male and female eye will lead to a better understanding of eye diseases and in turn benefit drug discoveries.

Read more from Health.AI:

Google DeepMind might have just solved the “Black Box” problem in medical AI

AI 2.0 in Ophthalmology — Google’s Second Publication

Deep Learning in Ophthalmology — How Google Did It

Machine Learning and OCT Images — the Future of Ophthalmology

Machine Learning and Plastic Surgery

AI & Neural Network for Pediatric Cataract

The Cutting Edge: The Future of Surgery



The latest news on artificial intelligence in healthcare

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store