NeuralNet Storyteller : Letting AI Write Stories based on Photographs

I have been into photography for quite sometime now, it was inevitable that my little brain would want to try something out with the photographs. So here is another little fun project I did to explore the combination of art and Machine Learning. I had developed a habit of publishing my photographs only if I could relate it to a story that strongly resonated with the photograph. This introspection led me into investigating if I could let AI write stories by looking at the photographs. And that is when I bumped on to the ULTIMATE NeuralNet Storyteller developed by Ryan Kiros et al from University of Toronto.

What does it do?

Given a photograph, it writes a story in the artistic style of a romantic novel without human intervention. Think of it as writing a passage for a romantic novel given a visually descriptive image.Before heading on to the technical summary, let’s look at the results of NeuralNet Storyteller on some of the photographs:

OUTPUT: My descent to the moon , blinking in the evening sky . It was as if I had spent the last thousand years trying to find a way out of it , but I did n’t know what else to do . Instead , I felt a sinking feeling in the pit of my stomach . The reality of the situation was so much more than that Elizabeth had caught up with him , and she simply held his arm out for her . The sun ‘s rays rose like stars on the horizon , enveloping us . I wanted to stay alive .

OUTPUT:
Images wing of the plane was spectacular , and I wondered what it was like to fly out of the sky . For the first time in millennia , I could n’t find the truth in the news . In fact , it had been more than ever since my plane landed on Earth . It was as if he breathed in and out of the air , looking up at the sky above us . The plane had begun to take its toll , but that ‘s the only way possible . I wanted to rescue her and hold my breath .

OUTPUT:
I bird barely touched the water as it was in front of me . By the time I reached the source of the fog , I could hear the rise and fall of his chest , leaving her gasping for air . I wanted to talk to her , but I had no intention of letting her go . In fact , it was the most beautiful thing I ‘d ever seen . The sun rose above the horizon as a bird swam in and out , leaving me gasping for air . It seems like the right person to ask who he is , that I had fallen asleep in the water .

OUTPUT: By what I saw a view through the rearview mirror , I did n’t turn away from the car . It seemed to be the most beautiful thing in the world . In fact , it had taken a lot of effort to control my body . By the time he pulled out of the car , I wondered if she was still alive , but I had no intention of letting her go . Her lips parted slightly , and she forced herself to look up from my rearview mirror as if that was the last time I d ever seen Gabriel . ‘

OUTPUT: By the time I pulled out of traffic , I saw red lights blinking rapidly in the darkness . It was driving me crazy , and I had no idea what it would be like to go back to Chicago . In fact , it was a lot more difficult to deal with . The city lights dimmed , flashing lights flashing through the streets of London as he drove his car into the alley . I did n’t know if that was the case , but she kept her gaze averted from mine and let the tears flow . “ As soon as possible , it s the only way I had a good night .

OUTPUT: People group on the beach , not really . There was no time to react to my own feelings , but for the first time , I felt as if they were standing on top of a sandy beach . I had no idea what she wanted to do with him , and he was the most beautiful girl she ‘d ever met . His sandy sandy beach was plastered on his back and his arm around her shoulders . Thank God , that would be the end of the season . I kept walking , oblivious to the reality of everyone else there.

OUTPUT: White made a wild move through the forest at the back of the clearing . By the time I knelt in front of him , my heart was growling , and I could n’t see anything other than the mysterious animal who had come out of the forest . I mean , I don t know what to do , she said . There was no reason to be as much fun as possible . By the time the sun had set , she was also going to walk through the woods and into the woods that surrounded him . He loved it , leaving Devon in such long strides .

OUTPUT: I pulled through the dark building , a window to the side of the building . It was a shame that I had no idea what it would be like to go out , but I did n’t know what else to do . In fact , it seemed as if my entire body had been on display for the last twenty-four hours . By the time I saw him enter the building , he gave me a false sense of satisfaction . The apartment building was a stark contrast against Eva s , and she wanted to tear her eyes open . So much of it , the building and a brick wall that could only count .

OUTPUT: I reopened the sky , expecting the sun to rise and fall to the ground . It was the most beautiful thing I had ever seen in a very long time . The sky was filled with the faintest trace of light , as if the air around us had gone out of my mind . In fact , it seemed like a million miles an hour , but she did n’t care . She shook her head and let out a small sigh of relief , because that was the only thing he could hope for . She was more than willing to stay in New York and under the radar , the sun gave way off .

OUTPUT: We had to make a group of people , both of them . For the first time in my life , I was pretty sure she had no idea what to say . She sighed heavily and looked up at me . In fact , the rest of the crowd was gone , and he seemed to be the only child he ‘d ever met in New York . There were so many people around who fell into love with each other , giving her a small tug of comfort . I m glad to know that , the people around them could barely stand and talk .

OUTPUT: Seeing the window to look out , I took a deep breath . It was as though I had never been able to find a place to enter the city . In the past twenty-four hours , I could tell by the flash of light on my face , I could n’t help but notice that Sofia was still beautiful , despite the fact that she had no idea what she wanted from him . It was more than a few windows , and it made perfect sense . My entire body shivered in anger .

OUTPUT: We drove steadily to the street at night , but I had no idea how fast it would be . The city lights dimmed , flashing lights flashing in the fading night . I did n’t know what else to do , so I pulled my purse from my pocket and pulled out onto the street . For some reason , I was more than willing to deal with him at the very least . In fact , it seemed as if she had been building up on the streets of New York City . She could hardly see the streets of Chicago , that ‘s what happened at night and at normal times .

OUTPUT: That city lights flashed at the street , though I did n’t have time to deal with it . The night before , I could see the city lights blinking rapidly in the darkness . The city lights dimmed , and I wondered what was going on in my head . For the first time in so many years , he seemed to be more than willing to keep his distance . She told me that she had no idea what she was feeling at night , but the streets of St. Giles ‘s penthouse apartment complex . Both of us , I could see the city lights and cast out at night .

OUTPUT: We had to act gathered together at the very least , most of the people in attendance . I was n’t sure what they were looking for , but it was more than a few short months ago . As soon as the crowd gathered around us , I let out a deep breath and hugged my sister . She was beautiful , and she had no intention of falling in love with him . In fact , it occurred to me that James and his men had made an appearance in the past century . She seemed to think they could do more than just one of those people , coming back and forth .

OUTPUT: No man in his car was trying to explain , so I took a deep breath . For the first time , I heard a woman call out to him as she exited the car . I had never been in contact with her , and I had no idea how long it would be . In fact , I could n’t help but notice that the driver ‘s license was gone now . My body warmed up , and I forced my head to the side of the parking lot , holding onto my cell phone .

And How?

The NeuralNet Storyteller takes an image, recognizes the objects in the image based on which it produces a caption and then transforms the caption into a short romantic story using what is called Style Shifting.

The only part that is trained in a supervised manner was for generating captions using Microsoft COCO data. RNN is trained on Romance novels to first build a decoder to convert passages from the novel to skip-thought vector representation. These skip thought vectors are then conditioned to generate the passages that were used to generate the skip-thought vectors.To obtain the artistic style of Romance Novel, this dataset (romance novels from BookCorpus) has been used. In order to embed new images and retrieve captions, a visual-semantic embedding is trained between COCO images and captions. The captions and images are mapped into a common vector space.
The Skip-Thought Vectors are obtained from an unsupervised approach to train a generic, distributed sentence encoder.Sentences that share semantic and syntactic properties are mapped to similar vector representations. These Vectors make it possible to construct a Style-Shifting function in a simple way that bridges the gap between retrieved image captions and passages in novels.

1. RNN is trained on Romance novels for encoding and decoding passages from romance novels to skip-thought vectors
2. Simultaneously, a visual semantic embedding is trained to obtain captions for given photographs
3. Input: A photograph is given as input
4. Obtain Caption for Photograph: The Visual Semantic embedding predicts the most suitable caption for the image.
5. Style Shifting: The caption is then translated into a romantic story type of passage by keeping the ‘thought’ of the caption and replacing the caption (descriptive) style with romantic story style.
6. Output: TA DA! A Romantic Story Syle Caption for the Photograph.

I hope you are as impressed with the results as I am. If you have ideas for fun AI projects or if you are looking to collaborate, give me a shout-out here or simply comment below. Also, checkout my photography work on facebook and instagram.

This write-up was originally written here.

Reference:

Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. “Skip-Thought Vectors.” arXiv preprint arXiv:1506.06726 (2015).
Code: https://github.com/ryankiros/neural-storyteller