Remembering Yahoo Pipes
Today Yahoo announced Yahoo Pipes will be shut down. Here’s what I wrote about it 4 years ago, for a book that was never finished or published, slightly adapted to make sense as a post.
It was part of a book about YQL (Yahoo Query Language) that GC and I took over in 2011.
The history of YQL
Before the creation of YQL, there was another product, called Yahoo! Pipes. This product provided much of the conceptual framework that became essential to come up with YQL. It was also very important because the real-world experience with users ended up exposing the need for a more advanced approach to the “Unix pipes for the web” initiative.
Unix pipes for the web
The concept of the Unix pipes is a very powerful one. Operations feed data into each other, executing loosely coupled processes in different programs to achieve a final result, that was “piped” through this chain. One could say it’s almost as a mashup is happening locally at your machine.
Tim O’Reilly wrote a blog post in 2007 trying to pin down who was it that had the Web Pipes idea, where instead of chaining together Unix commands, you could chain together data processed by different APIs on the web. He came up with references to both Jon Udell and Andrew Schulman, who talked about this at a Perl conference in 1997.
Everything that was needed to execute this vision was coming together. APIs were flourishing during the Web 2.0 movement, people were consuming RSS feeds and becoming familiarized with the concept of mashups. Mixing and filtering data was becoming a need most people didn’t have before. Finally, in February of 2007, all of this culminated with the creation of Yahoo! Pipes.
The creation of Yahoo Pipes
In the words of Kevin Cheng, Yahoo Pipes designer at the time, these were the people involved in the launch:
It was Pasha Sadri’s baby. Edward Ho worked on many parts of it. Jonathan Trevor did much of the front-end engineering though all three of them contributed to every aspect of it. Daniel Raffel was the product manager and ran through a lot of Yahoo! internal stuff to get it shippable.
I joined a couple of months after the project had started in earnest as the designer and, building on Pasha’s original Pipe interactions, reorganized and redesigned the site and the tool. The five of us were the launch team.
Kevin Cheng, “Who created Yahoo! Pipes?” on Quora
The Yahoo! Pipes website explains that the product is “a powerful composition tool to aggregate, manipulate, and mashup content from around the web”. And doesn’t leave any doubts to whether previous technologies inspired the service: “Like Unix pipes, simple commands can be combined together to create output that meets your needs”.
When Pipes launched, in February of 2007, some people were very excited about it.
Yahoo!’s new Pipes service is a milestone in the history of the internet. It’s a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. […]
It democratizes web programming, making it easier for people to have more control over the internet information services they consume, and providing a general-purpose platform for interacting with sites that is more powerful than the browser or feed-reader alone, but without requiring full programming skills.
Tim O’Reilly, Pipes and Filters for the Internet (Feb 2007)
In the screenshot above you can see how Pipes is highly visual. On top of the powerful ideas that inspired the product, the rich, fluid interface with the little hoses that connect one box to the other and transfer data between them were quite innovative and hard to do.
Hard core developers used Pipes, but understandably, the service also started attracting much more than just programmers used to the command-line. And this wide range of users became a big factor later, in the creation of YQL.
How YQL emerged from Pipes
Jonathan Trevor was on the original Pipes team and became head of YQL during some time at Yahoo!. He explained to the 2008 QCon SF conference audience what Pipes is (watch the full video of Jonathan Trevor’s presentation about Pipes and YQL for a great Web 2.0 throwback).
In the presentation, Trevor comments that the product team was having a hard time balancing the feature set in Pipes. On one side, more and more advanced features were required by advanced users, but on the other side, the service could not become too complicated for non-programmers, or they would lose interest in it.
Around 24 minutes into the presentation, Trevor says:
We did this thing of listening to our users’ desires. And our users had this breadth: we had the simple users, and we had the programmers, the über-users, if you will, at the other end.
And so, we started adding more and more functionality. We started adding more sources, […] then we started to add more and more modules, that did more and more complex things […].
To certain extent, this is a difficult line to tread, because you give the system more capability, but it becomes harder to explain, harder for people to use.
So, at a certain point, we started stretching the capabilities of our visual environment, and this is where […] YQL starts off.
Jonathan Trevor, Pipes and Y! Query Language
This talk was one of the first times YQL was shown in public and explained to an audience. At the time of launch, in 2008, there were no community tables and only `SELECT` statements were possible. But now the über-users had a full query language to work on, that was more suited for advanced uses, more flexible and extensible than Yahoo! Pipes.
Why does it matter?
We live in a recombinant world, in a remix culture. We build on top of what others have built before, and that is what moves us forward.
Recombination happens everywhere since the beginning of time. It is the driver of evolution. Copy, transform, recombine. It happens with or without human intervention, in biology, in music, in art, in philosophy and of course in technology, the area that today empowers all the others to be recombinant in a networked digital world.
When computers arrived, we started to understand that it was possible to share bytes and not lose them. It was not like lending a book. You could lend the book and keep it at the same time. With the Internet, these bytes became available all over the world. You could lend a book to someone that was thousands of miles away, and you still kept your own. Knowledge was multipliable, and had no more boundaries. This, in a nutshell, is the digital revolution.
Being able to create or reproduce data, modify it and publish it for others to read and modify is the core of the Read/Write Culture that Lawrence Lessig talks about. A culture where people are not limited to just consuming broadcasted messages, but have the power to create and share as well.
Take the human instinct to recombine and put it in the context of the Read/Write World Wide Web. It is only natural that hacking emerges as a trend so strong that starts driving businesses priorities and the direction of technology.
Ask anyone who has a young website that starts to become popular and you will see a pattern: there is always a group of users that are not satisfied, that want more or different things from the service. This network of unsatisfied users with some technical knowledge start to fix things regardless of the plans of the website being fixed. This is the hacker agenda taking place.
Hacker: one who changes a system so it works in ways that are different from what was originally projected.
When Web 2.0 hackers take your data and build something better than what you have, it’s hard to ignore. This is how companies awakened to the fact that there was a real need for opening up data and making hackers’ lives easier.
APIs are an answer to this demand. It stands for Application Programming Interface, a standardized and controlled way for developers to tinker with data.
Why expose data via APIs?
Cal Henderson, in 2007, puts it better than I ever could, explaining the Flickr API.
Opening up for developers helps the ecosystem grow, helps you understand user needs earlier.
If the platform or service provides value to end users, independent developers will work on it. There are many examples of this principle in action, external developers and the user community driving the roadmap by adopting third party solutions that work better: Instagram, Tweetdeck, Tweetie.
This is why it is called a web. No node stands alone, and the connection is greater than the sums of its parts.
Looking back, the examples above seem naive and idealistic, but hacking the open web was the reason many of us got involved with this whole business in the first place. Today, the open web as we knew it is mostly gone, and the demise of Pipes is a sad reminder of that.
