To Recognize Families in the Wild: A Machine Vision Tutorial (Part 1 — Overview)

Joseph Robinson
Nov 19, 2019 · 4 min read

Please checkout updated version,

My PhD research involves kinship recognition and our FIW dataset. Over the near future the goal is to summarize key findings, while providing lessons and demos of data challenges being held at top tier conferences annually (i.e., this is the first of many to come).

To learn more about FIW visit the project page. To learn more, register, and be involved with the upcoming RFIW 2020 visit workshop page.

The ability to automatically recognize blood relatives (i.e., kinship) through imagery holds promise in an abundance of applications. To name a few areas: forensics (e.g., human tracking, missing children, crime-scene investigations), border-control and security, dislocated refugee families, historic and genealogical lineage studies, social media, predictive modeling, and even as search cues for facial recognition (i.e., had we known the “Boston Bombers” were brothers).

Here in SMILE Lab, we are amongst the initial pioneers of the problem [6–9]. In 2011, we paved way with one of the initial public dataset for the kin-based vision tasks, UB KinFace database. Still, several years later, we fuel the “deep-learning” craze with the 1st large-scale image database for kinship recognition, the Families In the Wild (FIW) database [3, 5]. FIW provides imagery for 1,000 families, averaging 5.6 members and 13.7 family photos a piece; each family is labeled entirely with all member names, genders and relationships included — FIW serves multi-task purposes from this rich label information [1].

We launched the 1st large-scale data challenge for kinship recognition, Recognizing Families In the Wild(RFIW) [3]. Started in 2017 (i.e., RFIW’17) as a Data Challenge Workshop held in conjunction with the conference of ACM Multimedia. After a successful first RFIW, with 101 teams registered between two tasks (i.e., kinship verification and family classification), seven papers accepted in 2017 Proceedings on RFIW and presented at the workshop. For this, we also were honored to have such great keynote speakers: Chris Myles of DHS and a researcher Caiming Xiong of Salesforces.

Ever since, the RFIW has transformed into a series, with Challenges held in 2018 and 2019 held as IEEE FG Challenge [2]. Protocols, experiments, and models recorded as benchmarks and published as state-of-the-art are available to the public in our open-source repo. For a deep dive on the FIW data see our PAMI[1].

After 9 publications cited a total of 500 times, 2 datasets, and 4 Workshop Challenges, and a recent Kaggle Competition, we expect momentum to only pick up from here — researchers are attracted to RFIW, which has formed into an annual challenge. Besides, FIW, as a whole, attracts experts, new PhDs, and even professionals from other fields. Surely, the problem now has the needed resource attention need to transition from research-to-reality.


[1] Joseph P Robinson, Ming Shao, Hongfu Liu, Yue Wu, Timothy Gillis, and Yun Fu. “Visual Kinship Recognition of Families In the Wild” IEEE TPAMI Special Edition: Computational Face (2018). [paper]

[2] Yue Wu, Zhengming Ding, Hongfu Liu, Joseph P Robinson, and Yun Fu. Kinship Classification through Latent Adaptive Subspace, in 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE. [paper]

[3] Joseph P Robinson, Ming Shao, Handong Zhao, Yue Wu, Timothy Gillis, Yun Fu. Recognizing Families In the Wild (RFIW): Data Challenge Workshop in conjunction with ACM MM 2017, in ACM Multimedia Conference: In Proceedings of the 2017 Workshop on Recognizing Families In the Wild, Page(s): 5–12. ACM. [paper, proceedings, contents]

[4] Shuyang Wang, Joseph P Robinson, and Yun Fu. “Kinship Verification on Families in the Wild with Marginalized Denoising Metric Learning,” in 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Page(s): 216–221. IEEE. [paper]

[5] Joseph Robinson, Ming Shao, Yue Wu, and Yun Fu, Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarks, in Proceedings of the 2016 ACM on Multimedia Conference (pp. 242–246). ACM.

[6] Siyu Xia, Ming Shao, Jiebo Luo, and Yun Fu, Understanding Kin Relationships in a Photo, in IEEE Transactions on Multimedia (T-MM), Volume: 14, Issue: 4, Page(s): 1046–1056. IEEE, 2012.

[7] Siyu Xia, Ming Shao, and Yun Fu, Toward Kinship Verification Using Visual Attributes, in Pattern Recognition (ICPR), 2012 21st International Conference on (pp. 549–552). IEEE.

[8] Siyu Xia, Ming Shao and Yun Fu, Kinship Verification through Transfer Learning, in Proceedings-international joint conference on artificial intelligence (IJCAI). Vol. 22. №3. 2011.

[9] Ming Shao, Siyu Xia and Yun Fu, Genealogical Face Recognition based on UB KinFace Database, in Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on. IEEE, 2011.

Want a vaguely-named publication for your fiction, non-fiction, essays, prose, and more? Looking for an editor to challenge you? Want a chance to say, “Oh, I was just published… Yeah, That Publication!” Welcome to THAT PUBLICATION, your home for every story.

Joseph Robinson

Written by

That Publication
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade