I loved putting together the first issue of Near Future Field Notes. I knew what sorts of things I wanted to talk about, what sorts of things I wanted to try to avoid, and how I wanted to put it all together into a package. It was really fun seeing it come alive, and as I worked I ended up with enough notes and ideas to write several more issues. So then the challenge become sequencing.
If you have 20 different things you want to say, what should you say first? Are there natural combinations of things that you can pair together? Should each issue be grouped tightly to a theme, or should it be more loose? What do I want to say, and what might resonate with people? Is there any overlap?
These are all fun problems to work through and try to design for. But I knew one thing: I didn’t want to write one-off essays. I wanted each to hang together as a single work. In musical terms, I want to write each issue as a concept album, not release a series of singles. Even if it makes it harder to get air time.
I also wanted to be true to the concept of “field notes.” I don’t want to write from the point of view of someone who has answers. I don’t want to pretend I’m a thought leader. I want to be curious and take you with me while I learn things. Even if it doesn’t necessarily fit into a tidy theme, and even if it doesn’t make me look like I know what I’m talking about. Especially then!
I’ve been reading lots of books and learning lots of new things. But the most fun I’ve had since last issue, by far, is how to scrape data off the internet for further analysis and study. I want to explain how I did that. But to get there, I’ll need to explain why that became an obsession of mine. And to explain that, we need to talk about framing, formats, and grains. Let’s start there.
You can learn a lot about a designer’s eye by what sort of bad design frustrates them. For example, some designers are offended by use of bad typography. Others might notice laboured copy, or subpar use of white space. Others notice how many taps or clicks a user has to make, or the details of how quickly a page loads on the internet. Others might judge a product on how much it’s challenging the status quo versus sticking to well-worn solutions.
My eye seems to go straight to the “grain” of a medium. It helps me as a designer but it’s also deeply frustrating. If you show me an app on iOS, my eye will immediately discover and resent the non-standard patterns and elements used. For example, I can love Google’s Material design on Android but not on iOS. To me, it’s painfully discordant. It just feels wrong.
This is also why I struggle at conferences. I want the person on stage to understand that they’re speaking to a room of people, using a microphone. I want them to adjust for the fact that it’s an entirely different communication method. One with its own nuances, best practices, and grain. So when I see someone in that situation read from a piece of paper — a piece of paper that I could just be reading on Medium on my own time — it can feel like nails on a chalkboard. On the other hand, when they do well within the parameters of the stage — as many do! — it feels like a triumph. My heart sings.
Books are another example. Sometimes authors stretch their books much further than the content can justify. I call them “blog post books” and they drive me crazy. Authors shouldn’t write to hit an imagined page number, they should write until they’re about to repeat themselves. Then they should stop.
There’s this strange mythology about how big and bold bets drive innovation. But it’s not just about boldness. There have been plenty of bold ideas that didn’t pan out. There have been plenty of incremental steps that laid the foundation for enormous changes in society. The key isn’t boldness or aggression for their own sake. The key word is listening.
When you listen well, you design well. When you do your homework by studying trends and the actions of people, you’re able to see where things are headed. Then the next steps, whether bold or incremental, seem obvious in hindsight. Because they come from framing the problem correctly, which can only happen when you’re listening well.
Here’s an tangible example.
My family recently moved, and I wanted to write a diary of our experience so people could follow along. In some cases, people would be following along in real time. But I’d also want to be able to share our story sometime in the future. Sort of like a book where we’d add a chapter at a time in real time.
At first blush, it sounds like a job for a blog, Instagram, Facebook, or Tumblr. But in all of those products, the content is experienced in reverse-chronological format. (And often blended with other people’s content) This means readers have to sift through the most recent content to get to older content. That works pretty well for some scenarios like general news. But for documenting a big move, those products just don’t work. They’d go against the grain of the scenario I wanted to build. So I wrote some new software that was more like a book. It’s worked really well. I wouldn’t call that decision “bold,” I’d just say I tried to pick the right tool for the job.
Or what about conferences?
After a decade of being disappointed by traditional conferences, I discovered a long-running show in Chicago called Too Much Light Makes the Baby Go Blind. They do 30 plays in 60 minutes (each selected by the audience!) which is a totally insane idea that works surprisingly well. The funny stuff is funnier. The poignant stuff hits you in the gut. Everything is heightened because the medium makes it so. It made me wonder if I could do a similar thing.
So a year or two later, I tried a new approach on stage. Rather than speaking for 30 minutes about one topic, I spoke for 3 minutes over the course of 10 different topics. And like the improv show I was inspired by, I asked the audience to participate throughout by picking the next mini-talk I should do. It ended up being my favourite talk of all time, and the crowd seemed to like it too. I had finally found a way to do something better than what YouTube could capture and better than listening to someone read a Medium post. I finally found a way to go with the grain of a stage performance. I loved it.
Wait, isn’t this supposed to be a series of essays about the Near Future? What does “the grain” have to do with Machine Learning, drones, the attention economy, blockchain, and AR? Everything.
It’s not just the thing you’re producing, it’s how you’re presenting it. It’s not just the medium, or the message, or that the medium is the message. It’s more about picking the right tool for the job. Framing your approach in the best possible way, and knowing that sometimes that means you can’t use the same off-the-shelf parts as everyone else.
It’s something I think about a lot with this publication. I’m not writing a book, but it’s not a single essay. It’s too long to read quickly and too short to have much weight. It’s more like a zine. It definitely goes against the grain of how Medium works best. It might be an accident, or I might discover something new. Not sure which way it’ll go yet, but it’s fun to see what happens!
Chefs and Cooks
Tim Urban has this metaphor of chefs and cooks. A cook sees a recipe and is looking to faithfully recreate it. On the other hand, a chef is attempting to write the recipe in the first place. It means a lot more failure, but in time the chef might discover new things.
We’re chefs most of the time. Whether you’re doing basic carpentry, looking to buy a car, or are looking for gift ideas, you’re not looking to start from scratch. You’re looking to see other people who have had a similar problem and then customise a bit from there. And there’s nothing wrong with that.
But there are places in our life where we have the confidence to try a new approach. Maybe we already know basic carpentry and want to build a treehouse from scratch, customise our own race car, or try to paint a painting. This is the mindset of the chef. It’s more exploratory and higher risk.
Companies understand they have to grow and evolve to succeed, so they want to try bold things, like a chef. But they’re a business, whose job it is to make money. Meaning they’re risk-averse and cautious. They’d rather spend $10 for a guaranteed $11 return rather than spend $100 for the shot at a $1000 return. The less likely the return, the harder it is to justify.
So they end up being chefs, which is understandable and often smart. But that strategy only works for a while. There’s no growth in it. That’s where companies plateau, and it’s where people stop being inspired. It’s a dangerous place to be, and it’s something I’m trying to avoid.
Waterfalls and Lakes
There’s so much information in the world. More than ever before. But it’s hard to find the valuable stuff and tune out the noise. For me, it starts with news. I’ve followed current events closely for my entire adult life. A few years ago I wondered what I was really getting out of it. And I didn’t have a great answer.
For many years there was a belief that a well-informed society is what makes democracy work, and so it was the job of every citizen to stay up-to-date. In this view, people who read newspapers were engaged and good. People who ignored the news were assumed to be uneducated, uninformed, maybe lazy.
But that calculation relied on one really big assumption: that the media was reporting the truth, and that members of society were all reading from the same general set of facts. For example, you could argue that the Vietnam War was either a good idea or a bad idea. But core facts like the number of combat deaths were reported and agreed on.
Things have changed. Now there’s more information than there’s ever been, but also more disinformation. That means far fewer facts we can all agree on, which erodes our ability to honestly discuss, well, anything. I began to wonder if maybe taking a break from the news might be good for my mental health. What I discovered it wasn’t just good emotionally … I was actually getting better information overall.
How could that be? How could less data actually give you better information? Because with a day or a week of distance from the news, you get better stuff. You can’t fall for rumours and gossip. Things have a chance to settle down a bit before you take it into your life. I’ve been thinking of it like a waterfall. Do I need to stand right beneath the torrent for a drink of water? Do I want or need to be battered by high velocity news? Or am I ok downstream where things are slower and more mellow? Like a lake?
I’ve had a website idea for a long time — entertainment, one year removed. So you’d see all the excitement and hype around new games, albums, tv shows, podcasts, or any other entertainment format. But it’d be shifted by a year before reaching you. It would make your entertainment purchases cheaper and more vetted. Like a lake instead of a waterfall. Same stuff, less froth.
So what does that mean for thinking about the future and spotting trends? Isn’t it important to know that MIT just published a significant finding 3 hours ago? How can you keep up if you’re always behind? Can you really spot trends if you’re not standing right at the bottom of the waterfall? Yeah. You just need different tools that work better downstream where the knowledge pools. So that’s what I’ve been working on.
Living Off the Land
“Scraping” has been around for a long time, and it just means pulling raw data from a website. So let’s say you want to see the latest stories on The New York Times. If you load the NYT in a browser, a lot happens:
- A lot of media and images are downloaded
- I use a lot of cellular bandwidth
- Ads and trackers follow me
- My eye drifts to the most compelling media and headlines
Meaning I am paying more money, and waiting for more time, to be tracked, advertised to, and distracted. Don’t get me wrong: I’m fine knowing what the NYT considers to be the biggest story of the day. But I want to see it my way. I want to know exactly what I want to know, and I want to see it the way I want to see it. Here’s how I did it.
Install a scraper
In the past I’ve used Beautiful Soup and these days I use Headless Chrome. I wrote some code that says “Get every headline from the New York Times and put it in a simple list.” And this is exactly what I did. When I decide I want to read the New York Times, I don’t go to their site. I go to my personalised list.
Get a personal server
This part isn’t strictly necessary, but it’s nice. I have a server in the cloud that does these tasks for me and then hosts them as a website. That means I can access the results from anywhere in the world, hosted like any other site.
Configure cron jobs (or launchd)
You can tell a computer to do a thing however often you want. For example, you might want to run something once a day, or every 5 minutes, or every 2 seconds. I scrape data every five minutes so my personal news server is always up-to-date. Since I use a Mac, I use a program called Lingon to help me set up my launchd scripts. On other servers, I’d use a standard utility called cron.
Dream up what I want to see
This is the fun part. I can list a bunch of links in a row, like RSS readers used to work. I could connect to my favorite four news aggregator websites, group everything together, and highlight all the content I know comes from low quality journalists so it’s easy to ignore. And this is exactly what I did!
Or I could follow every single link, download the content, strip out all the ads and clutter, and put everything into a giant PDF or ebook for easy reading. Can you imagine reading story after story without having to load things in tabs, or wait for anything to load, or be distracted by links? It would feel more like reading the web like a book. And I did this too. It’s amazing.
Then I aimed these tools at some great email newsletters I learned about. I love a good email newsletter, in theory. But I don’t want to get a bunch of links in my inbox once a week. I want to scrape every single link mentioned in the last year, then put the results into an ebook. I want to read that ebook sequentially on the bus, like any other book. I’ve been working on that, converting this…
… to this …
… and eventually I’ll have some solid reading material without any of the drawbacks of web-based delivery. I have a lot to learn, and the current webpage approach doesn’t let me learn quickly enough. So I’m living off the fat of the land instead, and having lots of fun.
Sometimes I wonder if there’s a business plan in all of this. Yes and no. I think people would pay money to have someone bundle up their news for them if it were done right. But it’s besides the point. I’m doing this because it’s fun. I’m not doing it for money, which is a good indication of how much you really enjoy something. Would you do it for free?
The Jobs We Choose
Imagine if every job in the world paid the same amount, and it was enough to live on. So when you met a teacher, doctor, janitor, artist, barista, or CEO, you’d know they chose that job for something beyond money. You knew they’d have picked it for what the job offered them.
How would that change your view towards tech workers? The industry has a bad reputation for being short on insight and long on stock and benefits. I saw it firsthand. The industry is full of entitled children that could use more perspective. But what if they were paid the same amount as anyone? What then? Would that change how you see them? It would for me.
Because then we wouldn’t be dealing with pampered millionaires. We’d be talking about people who could work fewer hours, with fewer responsibilities, and less internal politics, and yet choose not to. Why? If you take money out of the equation, these are people who are choosing to sit most of the day, staring at screens, sitting through a punishing slate of meetings, only to ship products to the world that they know everyone hates on. What sorts of strange people would put themselves through that if the money wasn’t there?
The Twitter and Facebook people would seem even stranger than most. People hate Twitter and Facebook. To hear the online mob say it, these workers are destroying Democracy, protecting Nazis, harming the vulnerable, and failing at each new product idea. Who would put themselves through that much grief? Who would sign up to be that maligned and disrespected? There has to be a reason. So it has to be the money, right?
Except that Twitter and Facebook don’t pay more than anyone else in tech. You could make the same money at Uber, Lyft, Google, Apple, Microsoft, or many of the other big companies in the same field. Hm. So why would anyone work at Facebook or Twitter? It’s not because they’re paying more money, so what could possibility make all the failure worth it?
Can I tell you a story?
I worked at Twitter for several years, and for a year I was on the team working to help Twitter handle abuse. I presented a talk at the design team’s annual design day called Low Fidelity Is Magic. My main point was that designers should keep things low fidelity longer than they might think. To get my point across, I asked everyone to do a series of 4 frame comics. Later I talked to someone who worked on my team and he showed me a comic he had done about his new job at Twitter.
And that’s what I saw everywhere I went. Everyone cared a lot. You can argue we cared too much, so it made it hard to work together. Everything was dialed up to 11. Everything was complicit, nothing was good enough. It made it hard to stay on the team long. It made it hard to feel proud. People burned out fast.
Think to yourself how frustrated or angry you are at Twitter or Facebook. Now multiply that feeling until you can’t sleep at night. Where you grit your teeth when you think about the mistakes these companies have made. Where it negatively impacts your physical health. And that’s who I worked with at Twitter. That was me.
I didn’t meet many people at Twitter for a free ride. There are other places to make money. There are other teams doing cool things. And practically everywhere in tech asks you to shoulder less baggage. The people that still trudge through it aren’t doing it for the money. They’re the ones pissed off enough that they’re still trying to make a difference.
They don’t get everything right, I’m the first to admit it. But I can tell you that they care. I know firsthand. I worked as hard as I could, as well as I could, for as long as I could. And in the eyes of the world, my team failed spectacularly. Twitter still sucks at abuse, after all. But I’m proud that I tried. Not for the paycheck but because I thought maybe I could make things a little better. I wish I could have done more. I know you do too. I promise I did my best.
(Written five years ago, when Medium was new: 24 August, 2013)
I remember in the 80’s when many of us debated the value of BBSes. The acronym stood for “bulletin board systems”, which described the technology in plain terms but didn’t do a good job explaining the magical feel of it. It was hard to explain why it was so different, and in many ways, so much better than anything that came before it.
I remember in the 90’s when we were all debating networks like Prodigy, AOL, Compuserve, and Fidonet and tried to figure out what it all meant. Some things were the “real internet” and other things were “walled gardens” but there was something special going on that transcended all the categorisations and debate.
I remember newsgroups and the web and browsers and people using the phrase “high tech” to mean anything that involved a computer. We talked endlessly about what the implications of everything emerging all around us. A lot of people all made the same joke about having trouble finding the onramp to the information superhighway.
I remember being told to buy Netscape stock. I remember the day Yahoo! went public and I started hearing rumors about how the valley was exploding with wealth. I remember breathless articles in Wired about the new economy and NASDAQ 10,000 and watching the market seesaw 800 points in a day. I remember how the pre-millennium tension felt. I remember how exciting it all was.
I remember somewhere in the middle of all of this a company named Pyra made this thing called Blogger. I remember all the original zine authors turned webmasters taking a look, comparing it to the hand-written products we had built out of duct tape, Perl, and chicken wire. I remember a parade of television pundits and most people dismissing blogs as vanity plates for the web.
I remember the crash, and the layoffs, and how within a few years it all felt hopeless. I remember how the internet had failed to prove it could sustain businesses. I remember the feeling that we had lost. I remember Zeldman saying “The independent content producer refuses to die,” and doubting him.
I remember Blogger losing almost its whole staff. Maybe there was five people, maybe three. Maybe it was just down to Ev. I remember a fund-raising campaign to save it. I bought a Cafepress mug and sent a sappy email about how I’d do what I could. I still have the mug; its Powazek-designed logo is almost completely faded away.
I remember that by the time Odeo and Twitter came around, we had already gone through a few more cycles but the internet was very much back. I remember how much we talked about Twitter. It was intriguing but confusing. Why would anyone write a blog post in 140 characters? Did we really need another place to waste time online? Why not just use MySpace?
I remember all the froth and debate, and I remember thinking “there must be something there if people can’t stop talking about it, loving it, hating it, trying to understand it, analysing it, comparing it to other things, sussing out its business plan, and so on”.
I think I finally understand now. In hindsight it’s clear that none of these inventions were radically different or sci-fi inspired. They were all just new ways for people to read, write, share, and communicate.
So what’s Medium? It’s the next in a long line of ideas that tend to get over-thought and downplayed at the time, but make a whole lot of sense once everyone starts imitating them.
Medium is a place to read and write. Human editors pick the best content instead of letting algorithms do it. This leads to a higher quality experience than on many other sites, so the links get shared a lot. That’s it.
It’s not x-ray vision, and it’s not a personal jetpack, it’s just a modest idea executed very well. And in my experience, those are the things people have the hardest time explaining at first. It may take us a few years to fully understand how simple but unique Medium really is on the internet in 2013.
And one day in the future, after the next good communication idea is announced, people will scratch their heads about that too, unable to understand the new value it brings. They’ll say “I don’t get it. Why not just do this on Medium?”
Same as it ever was.