Data-driven storytelling based on photo metadata.
How Storyo is using our digital footprint and AI to create stories using photos and contextual data to power photo products.
The camera rolls on our devices are uncurated visual timelines of our lives. And they grow everyday.
As a result the records of our most precious moments are many times lost in endless scrolls or entangled with disposable or unrelated imagery and chunks of similar photos proof of our determination to capture the perfect shot. And more importantly the life stories that get trapped within the data these lost memories contain.
Along with the impact in photography, smartphones today are processing powerhouses allowing complex algorithms to run locally and very fast. They are also at the center of many other aspects of our lives, shaping the way we connect to others, the world, our bodies and assuming a pivotal role in our data trail.
The essence behind Storyo is about leveraging this data trail and the power of modern devices to develop tools that uncover the stories in our photos, find the shots, give them context and add other bits and pieces of information, such as maps and real world information, to deliver it all in a narrative that can be cherished, celebrated and shared.
A data-driven storytelling engine.
When you take a photo with location data you create a bookmark. You pin yourself to a moment in space and time. Add a few of those together and you can have a detail account of what you have been up to in a specific timeframe. Add a few more and you can detect patterns, your favorite locations, your trips, the faces that make up your world and start to build a profile.
In our native SDK for iOS and Android for Photo Books, auto detection of stories in any given device begins with a scan of the entire photoset. Unrelated or otherwise ineligible photos are discarded and individual stories are isolated. As a film strip being cut in individual scenes, the camera roll is sliced in unique stories.
As an example Storyo SDK takes less than one second on an iPhone X with 20.000 photos, to detect all the stories in the camera roll.
This process is detailed by Manuel Costa (Storyo’s CTO) in his article: “Automatic Photo Curation through Storytelling and Deep Learning”and as he points out:
“We have been always in favor of having stories organized chronologically to represent a timeline of the users’ camera rolls in a story perspective. This idea has been validated by our users, with near 40% of the stories played in our video app starting with a tap on an automatic story.”
All these processes run locally, on the fly and use all the already available ML dedicated hardware, ensuring performance, privacy for the user and no server costs. Three of the main outputs that are key to automate the narrative creation are:
- User profile: The system compiles a set of statistics about a user based on his/her set of photos;
- List of Stories: Each story is categorized by a set of criteria like Frequent Location, Calendar events, Travel, etc;
- Photo ranking and profile: Each photo is similarly categorized and is also analysed and ranked for technical quality, aesthetics interest, face detection, similarity and storytelling relevance.
Space-time narratives
In Storio’s automatic story discovery feature, every photo is clustered in a space-time structure we call the Tree of Events. This is created for each individual story and preserves the chronological order in wish the photos were shot. This clustering structure provides the backbone for Storyo’s storytelling capabilities.
Let’s take as an example a set of 100 photos from a trip to Italy and France. From the individual profile, created in the initial analysis of the entire camera roll, the system determines that all the photos selected belong to the same travel journey and groups separate photo moments from different countries in one single story including the first photo taken at the airport of departure.
The nodes in the Tree of Events can be space or time based and each node corresponds to a possible chapter in the storyline. In our example, the top 3 chapters are space-based (JFK, Italy and France). As we go down the tree, sub-chapter segmentation can be dominated by space based organizations (ex: Rome and Venice) and others can be organized by time (ex: Venice Day 1, Venice Day 2).
From the Tree of Events the system can output more or less detailed storylines by exposing or not, the nature of each node in the final layout.
Elastic storytelling
The numbers of possible chapters in a story is directly dependent on the space-time nature of the photos in each photoset. But whether they are shown or not in the final storyline is algorithmically calculated taking in consideration parameters such as the total number of pages available in the photo book or the desired number of photos per page.
This flexibility allows the same 100 photos in our example to be turned into a brief 20 page book with 3 chapters or up to an epic 100 page, 8-chapter photo book, but both depict the same storyline or experience.
Telling the story
To understand the way the storyline is exposed in the final layout we need to get familiar with the concept of Storytelling Units. They are the design and content elements that give context to the sequence of photos and convey the underlined narrative. They can take up a full page in the book layout, like a map introducing a chapter, or can be placed along the photos such as a caption next to an image.
They can be text based and derive directly from the metadata in the photos like a location name (ex: street name or the time a photo was shot). But can also result from the clustering in the Tree of Events such as location names or timeframes common to all photos in a chapter (ex: Paris or Day 1). Others can be visual elements and also derive directly from the metadata, like a map with a point representing a photo. But again can be generated by the Tree of Events such a map with a line connecting two or more points depicting a movement.
These Storytelling Units can come from external sources by cross referencing space and time data. It can be a map service, as we already covered, or weather, Facebook activity and others. Pre-defined conditions determine when an external source is added. For instance, the system can cross reference weather conditions of the travel journey with those verified at home and add information if a relevant difference is found as weather will then be an important part of the experience. Similarly if the user’s Facebook account is connected, not only username and profile photo can be added to the layout but the system can also cross reference activity in the social network with the time and location of the photos and determine if a relevant post is available to be added to the story.
Designing Photo Book Stories
There are 3 categories of Storytelling Units: Full story level, chapter level and photo level. The way that these different units can be used is fully customizable. They can be in fixed positions in the layout or can be parametric and can or not show up depending on the specific photoset.
The easiest way to understand how the system can be used to create a photo book travel story is to think of if as a storyboard. Let’s again refer to our example and for the sake of clarity go with the 3 chapter version of the story.
For the sake of this explanation let’s also introduce the concept of Entry. In the case of photo books, Entries are the individual pages and each one has a role in the storyline. Some serve the purpose of displaying photos, others may be intros, story summaries, chapters breaks, external data sources, etc.
In our example we start with the cover that displays the top ranked photo of the pack and a title that starts with “Trip to” because the system profiled the story as a trip. “Europe” is the top level location name for all the locations outside the “home” country. The title is complemented with a reference to the time frame of the photos and because this specific trip lasted a couple of month the generated “time” sub-title is “Summer of 2017”.
The cover entry is followed by the intro page that displays a visual summary of the trip in the form of a map with the path of the trip. Given that the user is logged in with their Facebook account, profile pictures can be automatically added to the layout.
Due to the nature of the first chapter (which represents home and has only one photo) there’s no chapter break page so the correspondent map is displayed on the photos page of the chapter. Next come the second and third chapters, and both have chapter break pages that display a map generated from the bounding box of the following photos with a label that is the common location name for the images in each chapter. In these cases is the country name, Italy and France. Then follows the photos pages that are configured to display up to 2 pictures per page and a map displaying the locations of the photos.
By design this layout also includes the introduction of a Facebook post if available. By cross referencing timeframe and locations of the photos with the user’s posts, it is possible to add the most popular one to the story representation. This is just an example of a possible layout for a travel photo book — other options may give less emphasis to maps, or replacing chapters brake pages with text labels along photo pages. Other external sources of contextual data can also be added like weather or stock imagery cross-referenced with the photos content.
More than photobooks
The Storyo app uses the same storytelling concept in the form of video. Users have access to automatic stories created from their camera rolls but they can also create stories by selecting a timeframe of photos or picking them one by one. The app then creates a video on the fly automatically. The user can edit every aspect of the video, add stock imagery and share it directly from the app.
Another breakthrough that the app offers is the ability to create group videos inviting others to add their photos with the same automation and easiness. In the case of group stories each users gets a story that can be different from the other users. As an example, if I visit a friend in another country, my version of the story will reflect the fact that I’m traveling abroad by displaying a 3D globe showing origin and destination. My friend’s version will be a home story and the video might not even show maps at all.
Creating the next generation photo book products
As smartphones became the main device we use to take photos, our habit of taking photos has grown exponentially. At the same time photography itself has become a widely used tool to take notes, remember things or otherwise capture something for future reference. Add screenshots, memes, messaging services and an unlimited storage cloud service and there is a massive amount of images and data at the tip of your fingers.
Our technology is here to leverage all that data to power the creation of memorable stories for people everywhere in a simple and efficient way by using AI, neural networks and every means at our disposal to make the process of memory creation as seamless as our own experience of living those memories in the first place.
