How Facebook sees your images- Automatic alt text (Image recognition at work)

Recently, I was browsing facebook on a slow internet connection and noticed something similar to this on an image.

I was curious and wanted to know what is happening, so i took my profile picture image and did a view page source->control+f (image may contain) to see what is going on, and this is what i got.

- 1 person, beard and closeup

Facebook could describe my images. I got excited and tried to throw in some more images.

Image may contain: 9 people, people smiling, people sitting

Image may contain: 1 person, outdoor, water, closeup and nature

Image may contain: one or more people

Image may contain: 1 person, playing a sport and text

However, I could find out flaws in certain photos-

Image may contain: 12 people, people smiling, text”… Hahaha caught you Facebook- Stannis is not smiling ( and it is 6 people).

The current Facebook’s image recognition could find the following to a certain level of accuracy

  • number of faces in your pictures
  • if you are outdoors or indoors.
  • Sitting or standing.
  • If you are smiling or not
  • Proximity to camera ( selfie or not)
  • text in pictures

Facebook say that they are implementing this feature for visually impaired people to use Facebook features. https://newsroom.fb.com/news/2016/04/using-artificial-intelligence-to-help-blind-people-see-facebook/ I think this is a pretty great feature and I am very excited to think that this might even help in describing movies and videos in future.

Some other applications of this feature could be classifying fotos as Google has recently done.

A lot of SAAS companies could spring up around this tech as a lot of eCommerces, fashion retailers could use this for categorizing their inventory based on image attributes and also detecting wrongly classified items. Applications are endless.

If you would like to know what your Facebook image’s automatic alt text is, continue reading