Summer Visions in Berlin
The third edition of the Summer School on Methods for Computational Social Science took place last July in Berlin, in a historic mansion surrounded by the beautiful nature of the Wernsdorfer Lake. This year’s edition has focused on research and methods for analyzing multimedia data. The organizers Claudia Wagner (GESIS) and Nicola Perra (Greenwich University) did an excellent job for the third time in a row. They put together a very balanced group of 40+ excellent students and a lineup of 9 speakers I was lucky enough to be part of. Here’s my summary of that exciting week.
Automated Image Analysis for Social Science Research. Andreu Casas (New York University)
The school kicked off with an excellent 101 tutorial on deep learning for machine vision by Andreu Casas, which featured an interesting glossary of deep learning terms “translated” to more familiar statistical terms recognizable by scientists across disciplines. Andreu is a political scientist who embraced machine learning as a tool to answer questions about societal dynamics. He told a story about the persuasive power of images in steering the political life of our communities, which calls for new ways of channeling this great power for social good. Andreu is working in this direction by trying to quantify the role of social media pictures in creating real-world mobilization and protests. Machine learning models could do that quite effectively but at the cost of a ludicrous amount of training data. That’s why Andreu is currently building new tools that could quench the data thirst of AI with far fewer data entries.
Machine vision in social computing. Miriam Redi (Wikimedia Foundation)
In 1966, Seymour Papert, who back then was a research associate at MIT, drafted a proposal for a summer internship project. The document was titled “The Summer Vision Project” and it planned to solve in the span of a few months the problem of automatically detecting the shape of objects in a picture. It turns out that a few more decades of work were needed to figure out a good solution to that problem. With this short story, Miriam starts recounting the history of Computer Vision, with all its ups and downs, until the recent deep learning revolution. Miriam wrote part of this glorious history with her seminal work in computational aesthetics (her survey on understanding visual interestingness has been published recently). At Wikimedia, she’s now tackling a challenge with even greater importance for mankind: modeling visual knowledge. Can we use images to promote knowledge equity? How do people learn from images? Can we detect images with high encyclopaedic value across cultures? Stay tuned for upcoming work that will try to answer some of these questions.
Social Media for Lifestyle Health. Yelena Mejova (ISI Foundation)
Until the 5G revolution hits us with its swarm of intelligent sensors, smartphones remain the most ubiquitous form of distributed sensing. The endless stream of geo-referenced multimedia content captured by those devices and continuously poured on the Web contains precious information about our feelings, behaviors, and even about our health. Since her early days as a researcher at Yahoo! Labs, Yelena has pioneered the science of mining Twitter data to measure how our lifestyle — and especially our eating habits — impacts our health. Bad nutrition is the direct cause of most deaths in high-income countries, yet it’s notoriously difficult to track at scale. Yelena explained how digital data can come to the rescue: mentions of food and drinks in tweets are predictive of obesity rates at country and regional level; body weight can be inferred easily from profile pictures; and machine vision can inspect food pictures to infer their ingredients and energy intake.
The Spirit of the City. Luca Maria Aiello (Nokia Bell Labs)
Geo-referenced pictures uploaded on social media are also excellent descriptors of the space we live in. Five years ago, motivated by literature in urbanism and social psychology, I teamed up with my two pals Rossano and Daniele to embark on a mission to measure the “Spirit of the City”: the intangible aspects that determine our experience of urban places. We used social media (especially Flickr pictures) to draw many street-level maps of intangible properties of space (beauty, smell, sound, ambiance, culture, … you name it), and collected them under www.goodcitylife.org. In my lecture I showed how these maps can predict real-world outcomes, facilitate the work of architects and city planners, and improve the quality of our daily habits. But I also talked about the challenges of studying space through the lens of Web data after GDPR and Chris Wylie’s confessions inflicted a critical blow to open research on social media data. A great opportunity lies ahead for young researchers to use “unconventional” sources of data for spatial studies. In this spirit, I discussed how we combined open medical prescription data with food grocery sales statistics to measure the impact of eating habits on diabetes prevalence for the whole city of London (with a cool viz by Tobi Kauer, check it out!).
“Seeing” Spatial Contexts at Scale in Urban Sociology. Jackelyn Hwang (Stanford University)
Jackelyn provided a fascinating overview on the origin and milestones of Urban Sociology, from the early work of William Du Bois and the Chicago School up to the flourishing of Systematic Social Observation research (SSO), an ensemble of techniques to capture sights, sounds, and feel of the streets beyond survey and ethnographic methods. As a sociologist who’s well-versed in using state-of-the-art computational methods, Jackelyn is working on bringing SSO to the next level by leveraging the power of Big Data. She provided a great example of how Google Street View images can be used to capture elements of physical disorder, decay and low maintenance that, in turn, are linked with a number of important societal outcomes. In particular, she is working on the problem of training computer vision algorithms to detect thrash left on the street — a seemingly simple exercise that, in reality, hides lots of hard challenges ranging from the collection of reliable ground-truth annotations to adapting complex neural network architectures to this specific task. The work is still in progress but she showed already very promising results for the cities of Boston, Detroit, and LA. Fascinating lecture and an excellent example of how social science can reach new heights with the help of computational methods.
Data Exploration Through Artificial Intelligence and Dataviz. Mauro Martino (IBM Research)
Mauro is a dataviz rockstar. He landed in Berlin fresh from a deluge of press releases praising his latest creation: AiPortaits.com. On the website (which has been taken down after receiving millions of requests), you could upload a photo of your face and get back a unique version of it that looks like an old painting. The Generative Adversarial Network technology he used is so advanced it can produce really stunning works of art — much cooler than traditional style-transfer approaches. In his lecture, Mauro presented a selection of his finest creative production of the last few years, including the visualization of the rise of partisanship in the US House and the Forma Fluens project, a piece of art that uses doodles drawn by millions of people all over the world to unearth our cultural differences as well our universal mental models. With a breathtaking series of amazing imagery, Mauro demonstrated the extraordinary persuasive power of data visualization and reaffirmed the centrality of dataviz in the cycle of Data Science. For those interesting in learning the craft, he provided a useful list of resources to start from. He finished with a word of wisdom: to learn how to create beautiful things you should “get exposed to beauty” first.
The Visibility of News Sources across Media Channel. Sandra González-Bailón (University of Pennsylvania)
The collaboration of social sciences with the computer and physical sciences has strengthened considerably in the last few years; that’s because some of the most important questions on societal dynamics cannot be fully answered without a combined approach. Sandra’s research is an exemplary case of how to these disciplines come together to create a full-fledged computational social science approach. Strong of her background in sociology, Sandra could put under tight scrutiny the existing models of collective news exposure as they are often backed exclusively by self-reported measures and disregard the broad diversity of news outlets. By analyzing large-scale browsing data of news websites, she could record for the first time at scale the discrepancy between reported exposure and actual consumption, thus discovering that the often-ignored underwood of low-reach news outlets plays a massive role in collective information exposure. By building audience networks from the interactions between users and news websites she also challenged the selective exposure paradigm: unlike in social media, the news consumption patterns of readers seem not to be polarized according to political stances. Her book “Decoding the Social World” is out and available on amazon.
From Warburg to Deep Learning. Maximilian Schich (University of Texas at Dallas)
The exciting and fast-paced technological innovations we have been experiencing in the last decades makes us forget that the rich cultural scene we bask in is a direct emanation of the talent of our illustrious ancestors. Max Schich — an art historian working at the intersection of information visualization, computer science, physics, and art — poured on us copious wisdom from the past, like a river in flood. His very inspirational talk stretched from Leibniz to Star Wars and touched recurrently upon the concept of Culture. To emphasize how much physical artifacts have a deep influence on our digital life, he proposed a striking parallel between contemporary multimedia collections (like Pinterest walls) and the Mnemosyne Atlas, a collection of symbolic visual artifacts by historian Aby Warburg. Max has worked extensively on characterizing the complex evolution of cultural processes, for example by reconstructing the network of mobility of notable individuals over the course of two millennia. But that is just a piece in the much broader puzzle of cultural complexity that he is relentlessly trying to complete, piece after piece. Max is working on a book that we hope to be able to read very soon for more enlightening lessons at the interface between history and technology.
Social Speech Processing . Björn W. Schuller (Imperial College London)
Björn represents one of those (rare) exceptions to the commonsense rule: “success and modesty don’t go hand in hand”. His colossal corpus of work on audio processing for affective computing and digital health is absolutely spectacular, and I was struck by the simplicity, candor, and humility he demonstrated while presenting it. Using a blend of acoustic and linguistic information, mixed together with cutting-edge deep learning, Björn builds all sorts of sound-powered applications. Just to mention a few: an impressive noise reduction technique, a laughter tracker, an emotion detector, and a conversational agent that can effectively deal with patients affected by depression. Björn works also in the private sector as the founder of Audeering, a company for audio intelligence technologies, yet keeping up the spirit of open science by making lots of code and toolkits for audio processing available for free (check them out: CAS2T, Opensmile, OpenXbow, iHearU, DeepSpectrum, AuDeep, end2you). Bravo Björn, way to go!
…and, of course, the students!
Last but not least, a big shout-out to all the students who participated to the school. They patiently attended all the lectures and gave heart and soul in developing their mini-projects (congrats to the winners and to those awarded with honorable mentions). Most of all, they where the first in line when it came to the most important activities of the week: drinking, dancing and diving. That’s how you build a strong foundation for the next generation computational social scientists.
Thanks again to Claudia and Nicola for creating this unique opportunity and for letting us be part of it 💗 #iwillbeback
