Waltzing Along the Continuum - A Story About Poetry, Music, & Machine Learning
How an experiment in writing morphed into a journey to understand the ways tools can bridge the gap between human and machine intelligence.
Backstory
A couple of years ago, a friend invited me to come join him out on the west coast to work for Capital One Labs. My wife and I decided the timing was right for a grand adventure, so we picked up our roots in Minnesota and moved to San Francisco. I joined a small cadre of Capital One data scientists working within an umbrella project devoted to “fighting the bad guys” — which included developing systems to detect credit card fraud. With a nod towards the complexity of this system and a deep respect for the domain experts already working in this area, one of our colleagues, Paul Melby, had the brilliant idea to tackle this problem as a fusion cell. The fusion cell model emerged within the military, where the analysts and technologists who act on intelligence are forward-deployed with the personnel who gather the intelligence. In this model, new tools can be created on-the-fly and new guidance can be given to the reconnaissance teams — creating a lovely feedback loop.
In our particular case, we did not just show up to build new predictive models or investigative tools, but to try our hand at the work of the fraud investigators themselves. We realized that this approach would yield domain knowledge that we could not acquire any other way. And when we started to build the tools, we would actually be building them for ourselves first, in order to support our nascent expertise.
During the same time period, I began a project of a more personal nature. I have a deep background in music, but that part of my life remained largely dormant in Minnesota while my wife and I raised our two amazing daughters. This past summer, with one daughter getting married and the other facing just one more year of college, I decided that I finally had the time to re-energize the artistic side of my life.
I was interested in returning to songwriting, but my Capital One travel schedule kept me away from the only place I have ever felt comfortable while composing: a piano. So, I decided to take on a poetry project instead — something I could pursue anywhere. I was inspired by the photography collection, “Chased by The Light”, by nature photographer, Jim Brandenburg. For this project, he took one, raw photograph every day for three consecutive months. So, the challenge I set for myself was to write one poem a day for 100 straight days. I chose haiku for their simplicity and their formal constraints. In addition, while I wanted to be in the driver’s seat of this project, I also wanted to introduce some elements of randomness.
But this was all very separate from my work as a data scientist.
Inception
Then, last fall, Capital One announced the opportunity for associates to host sessions at SXSW. It became clear that this could be a way to combine my passions inside and outside of work.
During my research to more deeply understand both the expertise and tools of fraud investigators, I’d had an epiphany. Though the activities take place in entirely different domains, some really interesting common ground connects people who investigate fraud and people who write poetry. Much of this is tied to the nature of expertise itself. I wanted to explore this connection further, which led to questions about the boundary within the human / machine marriage:
- How do we divide up the labor, leveraging the strengths of each?
- How do we design open experiences which leave room for creativity?
- How do we define the interface so that humans can both understand and trust what a machine has just done?
- How do we build feedback loops into this interstitial intelligence such that it becomes more efficacious with use?
So, I drafted my good friends, Keelan Downton and Chris Moradi to join me. We started to pursue the idea that we could learn a lot about ways to design for experts in one domain by exploring these concepts in another. While you hear a lot about machine learning and artificial intelligence these days, we wanted to experiment with the interstitial intelligence that supports these systems.
In fact, we wanted to focus on a new definition for intelligent systems: systems of networked people and computers where boundaries have been designed to amplify intelligence, support creativity, and foster community.
We used the act of writing of poetry to explore the ways simple choices can lead to complex systems which draw upon user knowledge to continuously improve with use. Of course, the best way to learn how to design tools is to build some and then put them into people’s hands– which is why we decided to create working software to support our SXSW 2017 session, titled, Fraudsters and Poets: Designing for Experts. We created a web-based application which tapped into some lightweight machine learning (Markov Chains) to ingest public domain poetry from the early 1900s and generate haiku poems.
Wanting to focus on the human / machine marriage, we provided two mechanisms for our audience to interact with this poetry.
First, we allowed people to simply rate the poems we generated: thumbs up or thumbs down. This led to a discussion about language, collective learning, and measures of similarity across users and poems.
Second, we allowed users to make edits to these same poems simply by swapping in new words or lines from dropdown lists. Here, we dove deeply into featurization — discussing how these metrics could be created, combined, and then deployed to make the system smarter as it was used.
Our session was a big hit and it drew folks from a wide range of backgrounds — from software engineers to people who simply enjoy writing poetry. Although attendees were automatically logged into the app using a randomly assigned name of a famous poet, we did introduce a social angle to the session by asking people to raise their hands when we called out name pairs who ranked poems in similar and dissimilar ways. The audience was impressed by the way just a few simple clicks enabled them to write haikus — demonstrating the power of constraints. It is truly incredible how you can create a small, self-contained world using just seventeen syllables!
Next Steps
I believe we were successful in creating a compelling and educational, communal experience at SXSW — one which showcased the power of simple choices. However, when it comes to designing intelligent systems, our app only scratched the surface of what is possible. There are a number of other techniques we plan on exploring, including autoencoding of features, interactive evolutionary computation, and the agent-based modeling you find in complex adaptive systems. We also hope to explore different user interaction experiences, including virtual reality and interactive chat.
In terms of applications, we also hope to bring these principles of interstitial design to other domains. There are many examples of automated systems which require manual supervision, and these are ripe for the same fusion cell approach we used for our work in fraud and in poetry writing. Stay tuned for Part #2 of this article, which describes our foray into the world of non-profits which are tackling the deep challenges which stem from poverty — using artificial intelligence and the arts.
DISCLOSURE STATEMENT: These opinions are those of the author. Unless noted otherwise in this post, Capital One is not affiliated with, nor is it endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are the ownership of their respective owners. This article is © 2017 Capital One.
For more on APIs, open source, community events, and developer culture at Capital One, visit DevExchange, our one-stop developer portal.developer.capitalone.com/
