My Heart Feeds a Series of Tubes
The spiritual precursor to the modern smartphone app was probably the “data mashup,” which had its heyday a decade ago, before the introduction of the iPhone. Mashups recombined data sets and information in unique and novel ways, like plotting unexpected things onto maps or warping RSS feeds. They seemed to be poised to go mainstream back in early 2007, when a not-yet-wholly-irrelevant Yahoo launched a curious new service called Pipes as a platform on which to construct them. It was hailed as wildly innovative at the time, though you wouldn’t know it from the unceremonious Tumblr post this past summer in which Yahoo announced that they’re killing it off.
The name was especially clever. Officially, it was a reference to the |||||||“pipe”|||| character, which is used to pass values between commands on Unix systems, but it also handily evoked Ted Stevens’ bizarre characterization of the internet as a “a series of tubes,” a popular meme among nerds. Most importantly, it was a perfect description of the goofy user interface, which made your hacking feel like a puzzle game along the lines of Tetris. You’d thread together little cartoon bubbles representing your data inputs and the various operator blobs that could read and change them, and at the end of it you might find you’d built a weird new web tool of some sort.
I wasn’t a programmer back then, but it was through building projects in Pipes that I first learned how to think like one: carefully, logically, very slowly. Even after I later moved on to “regular” code, the fun interface, ease of use, and free infrastructure made Pipes my first stop for little occasional helper widgets that didn’t really deserve a “full-fledged project,” whatever that meant.
I’d often react to a case of information overload by opening up Pipes and trying to find a way to pare it down. This means my account is now littered with little data processors which attempted to solve the problems I’ve had in my life over the past decade. I realize these are very strange artifacts to feel nostalgic about, but we don’t get to choose these things.
Peter.json
URL query parameter input > iterate through third-party aggregator services > join results > sort chronologically > output results
The most advanced Pipes project I ever built was meticulously visually arranged down to the pixel because it was impossible to understand otherwise, and yet somehow the canvas still stretched out to about four times my available screen size. Peter had hired me for some audio-related consulting since I worked at a recording studio, and in the process he asked me to explore Pipes on the clock on his company’s behalf and try to build them a small tool related to podcasting.
It went very well, and we began to pursue progressively larger projects. The first few were still in Pipes, but eventually they started pushing the limits of the platform, using tricks like carefully delimiting fields with special punctuation markers so they could be used to store multiple data points. Jumping through those hoops made me weirdly fluent in “regular expressions,” a baffling method of analyzing text passages using arcane character sequences which are all but impossible for most developers to memorize. The list of basic technology concepts I still didn’t understand at that point is appalling in retrospect, but by golly, if you ever needed a script to cleanly extract phrases from a larger document, I was your guy.
The Pipes platform was rapidly increasing in sophistication with the introduction of additional processing bubbles, and I was mostly keeping up. One day Peter surprised me by offering to hire me away from the studio to work full time as a programmer, even though I didn’t know any of the languages his company used. I jumped at the chance: Their projects were strange and interesting, and the office was in a loft above a barn out in the country, into which they had installed a very nice recording studio of their own, complete with an elaborate and ecologically friendly LED lighting array which drew so little power that we’d sometimes run it during the workday just to choose a color that could set the mood for our coding.
Eventually the Pipes project turned into something resembling a graphical realtime search engine, a feature that Google wouldn’t even start to explore until three or four years later. Unfortunately Yahoo didn’t look too kindly on me trying to run four screens worth of processors at once (I’d bought a huge new monitor shortly after I got the job) so they swiftly and silently demoted it into a service tier in which it was only available every few hours, instead of continuously like most smaller Pipes. Our hand was forced; we began migrating pieces of the project into custom PHP code, which we could then run on our own servers whenever we wanted.
I was then asked to come up with a system for formally code-naming the release iterations, so that each major milestone would have an identifier, the same way most versions of Apple’s OS X were named after various wildcats. I went with cryptozoological creatures, those absurd not-quite-mythological animals like Nessie and Sasquatch whose existence can never quite be definitively disproven, because I had especially loved a big book about them back when I was a kid.
I worked for Peter for several more years, learning a lot while watching a few rounds of more senior developers come and go, including one unexpectedly affordable wizard who was in the process of changing his name to escape his well-documented past as a Holocaust denier. The company’s initial experimental weirdness faded away as financial pressures built, and eventually the barn-dwelling mad scientist artist geek collective started to feel like a more conventional web development agency. I left shortly thereafter. I’d turned into a programmer, and my initial Pipes prototype had turned into a functioning custom code library. It was shelved indefinitely after version 3.0, Mokèlé-Mbèmbé.
Hawes.json
news feed inputs > match predefined keywords > output results > filter out false positives in Google Reader
I do not recommend trying to get by in a small suburban town for several years without a car. My logic was that I couldn’t afford one, but being a pedestrian can be brutal in a town that isn’t really built for them, so instead I felt compelled to move into increasingly expensive apartments closer to the town center. I ended up living smack in the middle of the charming cobblestone downtown mall, and since I had no way to leave, I spent most of the next two years there, wandering in endless ellipses around an area roughly equivalent in size to four New York City blocks, looking for anything new and interesting.
I was working at the recording studio during the day and freelancing for the local newspaper after hours, mostly arts features and fluffy human interest stories, but was invited to start writing for their news blog after being the first to notice that some boring new restaurant had suddenly opened up in a particularly obscure cranny. The editor was an old school journalist who was doing his best to adapt to the internet. He once awkwardly published a sprawling investigation into a breaking murder case on a blog that could barely support it. The main site wasn’t scheduled to update for another few days and he sensed that people in town really wanted to know.
In his infinite enthusiasm for all things digital, he noted that my workload would be just as big no matter how he ultimately decided to publish the results, so he offered to pay me the print rate for blogging. This was an unreasonably lucrative offer. When that meeting ended, I raced back to my apartment.
The project with Peter was still in its infancy at this point, but I had already become a pretty competent Pipes wrangler in the process. We didn’t have reliable internet access at the studio, which made it very difficult to line up all the little pieces required to run any sort of after-hours writing career. Instead, I wrote a tool which would crawl across the internet during the long stretches when I couldn’t, looking for our most search-friendly local terminology — people, locations, bands — and spit all the results into Google Reader, where I could evaluate them later.
Every day I’d come home from the studio after work and eat dinner sitting in front of my computer, where I’d have hundreds of potential story leads stacked up for consideration. Many of these were false positives – but until it was killed off much like Pipes, Google Reader was one of the most effective productivity tools the internet has ever known. Using its key commands, sorting through everything only took about half an hour. Then I’d get to work, pitching some stories and just writing and filing others without waiting for approval, and keep at it until I found myself nodding off mid-sentence, usually around 3am. Sometimes I’d turn in as many as five stories a night; clickbait churn as we now know it didn’t really exist yet, so this pace only registered as exciting. I wrote sixty percent of their news stories that month — I still remember the number so clearly because I kept a running tally. Everyone was thrilled with me. To this day, one of the best compliments I’ve been paid by an editor is “You write faster than I can read!”
When payday finally rolled around at the end of the month, it was a bloodbath. I couldn’t stop giggling. Hawes sheepishly asked if he could retroactively lower the rate; my eyes rolled right out of my head and bounced down the hallway, and we both had to chase after them! He did dramatically lower the rate going forward, though. I shrugged and told him I’d just go back to the arts beat. He didn’t really like that answer either.
Hawes had once offered to save my ass with health insurance right when I needed it most, which is the sort of thing that will earn you my undying loyalty, so I told him about my secret tools and freely turned them over to the newsroom. The other reporters didn’t really seem to get it. Hawes saw the potential but couldn’t keep up with the volume of results. He still had a newspaper to run.
That would change eventually. The business side of the company started pressuring him to decrease operating costs by cutting one of the reporter positions on his already-small team. He refused, as much because it would hurt the quality of the journalism as because a person’s livelihood depended on it. But he finally found a way to give the business guys the headcount they wanted: he resigned as editor-in-chief and let them buy back the partial ownership stake he’d been given as a co-founder.
This was a shocking turn, since the paper had been his baby since the beginning. I no longer worked there by this point, but I dropped everything and called him the moment I saw the announcement, and he picked up on the first ring. By way of explanation, the best he had was “I realized this was probably the last time anybody was going to offer to pay me for a newspaper company.” Nine months later, our paper was absorbed by its principal competitor and shuttered for good.
Jamie.json
Omar.json
Daphne.json
real estate listing inputs > normalize across sources > scan text for important details > transplant location and price to prominent header position > output results
No matter how excited you are when you first move to New York from a small town, it is also terrifying in a way you never really get over. These are rents that can destroy you financially for the rest of your life if something goes wrong, and we are expected to just live right in the middle of them? It’s at least an order of magnitude worse than even the coolest downtown apartment in suburbia. Pipes couldn’t make it cheaper, but outsourcing the search to a customized processor for real estate listings did simplify the logistics of trying to find a place to land. More importantly, it acted as a coping mechanism, because it felt like I had a magic feather.
It worked incredibly well. Yahoo again hit me with a mysterious processing delay, but even with an extra hour or so inserted, I was still always the first one to respond to any given ad or listing, since I had a stronger signal-to-noise ratio than the regular craigslist interface. I found an awesome apartment share on the Upper East Side, which felt like an exciting place to start, if only because I’d seen a few reruns of The Jeffersons as a kid was aware of the theme song.
My roommate was sort of a nut, but the place itself was great! There was a gym, and a sweet roof deck, and a pool, and a doorman, and laundry machines in the basement, and all sorts of lovely amenities that I will never be able to afford now that I finally understand the proper amount of existential terror one should feel when living in the most expensive city in the world. Sara split her time evenly between working in finance and as a personal trainer. When the banks started collapsing in 2008 she immediately broke the lease at a five-figure penalty and moved back home to Florida, candidly accepting that she had no prayer of working on Wall Street while so many experienced full-time bankers were also out looking.
I just fired up Pipes again and found a new place in Brooklyn, then kept hopping between sublets until that got tedious enough to make me finally cough up the security deposit for a lease of my own. To this day I still live in an apartment I owe to Pipes. The realtor gouged me on her fee, but when I tried to push back she wasn’t having any of it; she just smirked and held firm, because she knew she had a good deal and I’d be taking it. That was five years ago.
Over the next few years, whenever any of my friends moved, I’d clone the real estate tool and lightly amend it to match their requirements. By this point I was also savvy enough to route the Pipes data through my own web site, a useless minor presentation detail which I always took care to implement because it made them all think I was much smarter than I actually was. Looking back at the list now, it was always the Pipes for old friends from other phases of my life who were just moving to town for the first time that were the most elaborate. I was so excited to see them again.
Brendan.json
blog feed input > pass filter if column name matches > output results
I’ve only really known Brendan from a distance until very recently, and for most of that time he sort of scared me. Before the merger, he was my rival on the arts beat at the competing paper, and he was unbelievably good at his job. I’d get nervous whenever I’d see him at a concert and start scribbling furiously, briefly convinced that robust note-taking would be the key to making my review better than his. We were perfectly cordial in that absurd way frenemies can be, but I always hated that we’d been cast into such a contentious relationship by our feuding employers.
I liked his writing enough that I kept loose tabs on his career for years even after I moved away. Eventually he moved to New York too, and began writing an online column I especially enjoyed. The site didn’t have individual feeds for the authors, so I had to use its busier master feed as an input to a simple Pipes setup which filtered out everything else and sent me notifications only for his occasional articles.
When he ended the column, I finally took it upon myself to email him and propose meeting up for drinks, but alas, I’d just missed him. He’d moved again a few weeks prior, this time to Montana to study writing — which was too bad, but I was happy about it because he really deserved to be a proper writer instead of just a journalist. He wrote back inviting me to stop by if I was ever out that way.
I doubt he ever expected me to take him up on the invitation, but I’d decided to leave my job right before the snowboarding trip, so I had nothing better to do than roam around the rest of the state before flying back, visiting old high school and college friends who had all ended up there. I’ve always hated driving, so I meticulously planned an elaborate bus route. At one point it required a trip on a tiny and relatively unknown shuttle service operated by and for a local Native American tribe, because I was outside even Greyhound’s boundaries.
I panicked when those precipitous plans began to unravel. I nervously hovered over my computer and almost just went ahead and bought an expensive plane ticket back home instead of a bus ticket into the mountains, but Ella encouraged me to keep pressing on, trying but totally failing to suppress a stern look that questioned whether I had any capacity for adventure. She’d just beaten back a devastating illness and then had a baby, so what was my excuse?
Brendan responded to my frenzied emails right away, but the next leg of the trip was awful. The bus departed at 3a.m., though the dispatcher told me it was regularly delayed up to six hours without warning, and the station was always locked overnight so there was no place to wait except out in the snow, which was piling up just rapidly enough to be disconcerting. The kid working at the gas station in the next lot didn’t mind three or four of us crashing his lonely late shift, but he said we’d have to stand out of view of the security cameras or he’d get in trouble the next day.
The mountains were gorgeous and impossibly huge even for mountains, like they’d been deliberately planted there as sculptures. I was a mess when my bus finally pulled in the following morning, but Brendan was already waiting at the station, even taller than I remembered, though that no longer seemed intimidating. He scooped me up with a smile and took me out to breakfast. We spent the rest of the day wandering, through fond remembrances of all our mutual friends, then out and around ponds frozen so solid you’d never have believed they were once made of water.
That’s the best of my archive — the rest is now incomprehensible, even to me. Pipes dies tonight, but you can download your projects until midnight. They’ll be exported in a data format called JSON, which for now can be read into nearly any other major technology system in the world; as small and niche as it was, even Pipes itself had a JSON input.
Even so, it’s hard to know what to do with these files after the platform that created them goes dark. They are just specifications for abstract filters that were built in a proprietary environment. What good can they possibly be if there’s nothing passing through them anymore? Maybe the answer is that all the processed information had to end up somewhere, and I still remember the best of it; the real data store is whatever we take with us. I’ll miss having the ability to quickly create smarter windows into the world that offload some of the exhaustion of simply trying to exist in it. At least we still have our own built-in Pipes, inputs like eyes and ears flowing into brains which struggle under the load and take way too long to make sense of it all, but in a pinch they’ll have to do.
