What I Learned from Feeding Dozens of Philip K. Dick Novels into a Recurrent Neural Network.
Or, perhaps, how I resisted the urge to name this article “Do recurrent neural networks dream of Philip K Dick?”
This piece is a chronicle of my developing curiosity regarding machine learning, with Philip K. Dick’s novels and short stories as the input data.
I am going to delve specifically into generating new text based on these novels using Markov chains and recurrent neural networks, there are links scattered throughout should you want to replicate or learn more about anything I have written about.
This piece is my addition to the rather niche topic of applying machine learning in relation to Philip K. Dick. In this endeavour I am aware that I am retreading some of the steps others have taken already. In particular aspects of the mildly unnerving, uncanny valley dodging Philip K Dick Android Project and in a case of convergent thinking, Aaron Souppouris’ K.Dickbot, among others.
Philip K Dick and his writing have captivated me ever since my first exposure. When I have recently become interested in the process of taking existing text and, using these words, generating “new” text based upon it using machine learning, his works topped my list for potential source material.
It wasn’t just PKD’s prodigious output that made him a logical choice for these machine learning exercises, it was the very themes that run throughout his writing that made the idea so attractive to me. It just made sense.
In Do Androids Dream of Electric Sheep we see the Turing test echoed in the administration of the Voight-Kampff methodology of detecting replicants, to determine if the responses given were from a human or something altogether artificial.
This need humanity has to decisively distinguish the artificial from the organic is a thread that weaves its way through many other of PKD’s stories, The Second Variety springs to mind as a particularly good example. The authenticity of the thoughts, feelings and ideas that humanity’s creations have are dissected, debated and killed over.
Concepts that Phillip K. Dick explored many decades ago in stories such as The Penultimate Truth seem so prescient now, when we think about carefully coordinated and automated Twitter bots influencing discourse by responding based on keywords or changing which topics are trending day to day. We live in an era of increasingly hard to detect computer generated “deep fake” images that can be created of anyone, of the human sounding, bland voice by design Google “assistants” making small talk while they book haircuts for their user over the phone.
For the somewhat simplistic purposes of this article, Markov chains generate new text modeled on existing text by randomly choosing successive words based upon which words follow each other in the original input corpus.
Below you can read, in the quoted text, some of the output that resulted from running the wonderfully easy to use Markovify by Jeremy Singer-Vine and markov-novel Python packages against a merging of the various PKD novels into a single text file.
When I first read about machine learning and text generation I was struck by the similarity of some of the resulting output to the “cut-up technique”. This technique of cutting up and rearranging existing printed text was pioneered by the Dadaists but is more widely associated with more recent writers and artists, in particular William S. Burroughs and David Bowie.
He stepped aside, then, and he gazed at her, augmented his salary at some earlier time, crossed deep space between planets in terms of raw percept-data, uncontrolled, unstructured, without meaning or its eyes, almost a million miles out.
I especially like “crossed deep space between planets”, “uncontrolled, unstructured, without meaning or its eyes, almost a million miles out”, it immediately conjured memories of PKD’s A Maze of Death.
What struck me most upon reading the output text was not occasional patches of gibberish or jarringly awkward sentence structure in parts, but that there was a surprising amount of it that was readable in a non headache inducing way.
I form one government and it’s a disturbance interfering with the bizarre and ubiquitous. Father turned it around for you, you would have bought land in New York.
The tone and the slightly weird sentence structure here wouldn’t necessarily feel out of place in Ubik or Martian Time-Slip. I should add that none of these excerpts have been edited internally in any way, although I have cut surrounding text in most cases.
And this is the day the authorities are trying to get electroshock equipment; they were almost finished spraying his face, making him clap his hands together.
Out of all of the text generated by Markovify I feel that this sentence above is the most evocative of the surreal combined with a palpable yet vague sense of foreboding that characterizes some my favorite PKD novels.
Next I progressed on to running the input text through a recurrent neural network, specifically the Python textgenrnn module by Max Woolf. I implemented this module on Google Colabratory to speed up the process, once again thanks to Max Woolf who wrote an excellent piece on availing yourself of training your RNN on a free GPU.
Deep technical explanations are definitely beyond the scope of this article so the Wikipedia explanation with a little added detail is going to have to suffice.
“A recurrent neural network (RNN) is a class of artificial neural network where connections between nodes form a directed graph along a sequence. This allows it to exhibit temporal dynamic behavior for a time sequence.”
The weakness in using Markov chains to generate text is that they do not maintain a long-term state. When you generate text with quotation marks or parenthesis, for instance, the Markov model will not have any way to remember to close them.
“Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs.”
This is an extremely brief explanation, the only two concepts relating to RNNs I want to cover from here are the epoch and the temperature.
Each epoch represents one entire pass through of the input data, with text that can then generated according to different temperatures. The temperature determines how optimal the predictive model will be, either more conservative at temperatures such as 0.2 while the closer you get to 1.0, the more “creative” the resulting text will be.
This text is from the very first epoch, with a temperature of 0.5:
and another pieces had already seen the control step in the boy of the house. “I don’t know the word with the contracted person in his own death and arrangement and slightly and growling from the seat, and then the forces of rational writer. He is the most problem that he did not still surge
Only “rational writer” really jumps out at me here, but this is just the beginning of course.
We reach epoch two, this time with a more conservative temperature of 0.2, a lot of big concepts covered in this rambling sentence.
The AI voice is a soul of the same time and the man in the same time to the state of the world
Epoch three, temperature 0.2 gives us this excerpt:
speech of the propose of the theory of the police state of the process of the past the property of the man with a strange thing that he was going to be able to see
Finally, Epoch 5 at temperature 0.5.
I wanted to be made out of sight and can be a business of midnight for a long way and silently made of what was looking for the first time the psychotic to the end of the end of the graves of the bed.
“A business of midnight” sounds like a novel title that wouldn’t seem out of place in an airport bookstore, I like it.
This experiment unfortunately ends at epoch 5 for now.
Leaving my computer running Colaboratory for any length of time created problems. Disconnections of any significant length cause Colaboratory to hang, necessitating starting over.
Then there is my cat.
Caleb wants to be good, I genuinely believe that, yet he has an irrepressible fascination with technology. This fascination, and desire to interact, with technology is unfortunate though, as Caleb has very sharp teeth.
In the last few months he has destroyed three USB keyboards, two auxiliary audio cables and a USB mouse, nothing with a cable is safe from his unwanted attention.
So what did I learn? I went into this whole enterprise rather sceptical about machine learning and with low expectations, the results from even just my minor dabbling impressed me though.
Will Markov chains or recurrent neural networks replace authors any time soon? No, but to think that they would at this early stage is missing the point.
In the future I want to take the time to run many more epochs of a recurrent neural network to see how the output evolves, unfortunately that will have to wait for now.
Also, perhaps most importantly, I learned to unplug and hide my USB peripherals whenever I would be unavailable to protect them from my kitten’s constant and voracious USB cable predation.
Working on this article I got to thinking about writing short drabble style stories that combined Markov chain generated text with my own. This is something I plan on doing soon, as I am already thinking about potential criteria and a framework for the writing process.
Finally, for those who would like a list of the Philip K. Dick works that were used in the creation of this article, they are:
A Maze of Death, A Scanner Darkly, Clans of the Alphane Moon, Confessions of a Crap Artist, Counter-Clock World, Deus Irae, Do Androids Dream of Electric Sheep, Dr Bloodmoney, Dr Futurity, Eye in the Sky, Flow My Tears, the Policeman Said, Galactic Pot-Healer, Gather Yourselves Together, Humpty Dumpty in Oakland, In Milton Lumky Territory, Lies Inc, Martian Time-Slip, Mary and the Giant, Nick and the Glimmung, Now Wait for Last Year, Our Friends from Frolix 8, Puttering About in a Small Land, Radio Free Albemuth, Solar Lottery, The Broken Bubble, The Collected Stories of Philip K. Dick, The Cosmic Puppets, The Crack in Space, The Exegesis of Philip K. Dick, The Game Players of Titan, The Ganymede Takeover, The Man in the High Castle, The Man Who Japed, The Man Whose Teeth Were All Exactly Alike, The Penultimate Truth, The Shifting Realities of Philip K. Dick, The Simulacra, The Three Stigmata of Palmer Eldritch, The Valis Trilogy, The World Jones Made, The Zap Gun, Time Out of Joint, Ubik, Voices From the Street, Vulcan’s Hammer, We Can Build You, What if Our World is Their Heaven
My personal favorite from that list is Clans of the Alphane Moon, if you haven’t read it then you should check it out.
