The many flaws in the New York Times take-down of Amazon

The anecdote-based and slanted article should have us asking questions…about the NY Times

*Disclosure, I’m an employee of Amazon. The postings on this site are my own and don’t necessarily represent Amazon’s position.

Last weekend, the New York Times had scathing critique of culture at Amazon, describing it as “a bruising workplace” where employees are supposedly on call 24-hours a day and spy on each other through anonymous feedback. The piece has attracted significant attention and generated over 5000 comments, the highest number ever. By those measures, the newspaper has succeeded in capturing the media cycle, clicks, and page views — which is how online news measures itself. By modern news standards, the piece was a hole in one.

But, was it good journalism? If you think journalism should be based on objective and data-driven analysis, then absolutely no. The Times built a slanted narrative from a few isolated, out of context anecdotes, that were pulled from a minuscule set of evidence. Many other journalists and media critics have questioned the article’s reporting — even the Times public editor said “the evidence against Amazon, while powerful, is largely anecdotal, not data-driven. And anecdotes can be used and interpreted in any number of ways.”

As an Amazon employee and former journalist, I was shocked by the article’s striking disregard for fact and objective analysis. At a high level, here are the top areas where the Times went off the rails.

  • Narrow set of potentially biased evidence.

For the article, the reporters spent months interviewing current and former employees. But note the number of interviewed: “more than 100 current and former Amazonians”, or less than .05% of Amazon’s 180,000-person workforce. While I applaud their effort to gather lots of anecdotes, in the end, 100s or even 1000s represent a relatively tiny and insignificant sample for a company as big as Amazon — and these interviews were the only evidence for the piece. That’s crazy! Such scant evidence will by definition be narrow and isolated.

To put this in context, at any company of 180,000, or 18,000, or even 1800 employees, it wouldn’t be very hard to find a handful of current and former employees who had horrible managers and horrible experiences. Statistical outliers almost always exist. But they should never be used to form a foundation for an argument. For proof of how a narrow set of evidence can be lead to various conclusions, see a follow up article by Fortune with interviews of more Amazon employees, which concluded: “Perhaps the irony of the Times account, is that when I talked to Amazon employees, most of them spoke with pride about the company’s culture”.

Not only was the amount of evidence insufficient, a lot of it could have been biased, which the Times never pointed out (Again, why???). Of the 100s of interviewers, many were former employees who left the company after horrible experiences, and their accounts, while perhaps true, should be red-flagged as potentially biased.

I’m not suggesting that the Times interview 1000s of employees (although that would be one step in the direction of statistical significance). But why not seek support for its striking claims with a more robust and reliable data set? How about the 5000+ anonymous employees reviews on Glassdoor? (Amazon’s rating is a 3.4/5, compared to Apple 4.0, Tesla 3.4, IBM 3.1). With not much effort from their data science team, the Times could have done a text-based statistical analysis on those 5000 anonymous reviews from former employees. They also could have used employee satisfaction surveys, employee retention data, etc? Anything more than just a handful of interviews.

  • Statements without context.

Several times in the Times articles, strong statements are made, but little to no context on the prevalence was given. For example, the article posited that “Employees say [the feedback tool] is frequently used to sabotage others.” But how many employees say that? A few, 10, 100…1000? I’m an employee, and I’ve never heard or said that, so it must not be that many. The same problem arises with other statements like, “emails arrive past midnight, followed by text messages asking why they were not answered”. Again, I’ve never heard of that happening. So, is that 1 person’s experience, 10 people, 100? When these statements are given out of context, they’re rendered meaningless.

Many others have also pointed this out. In a critique of the piece, longtime journalist and media expert Jeff Jarvis wrote that the article “lacks two key attributes: context and — I can’t quite believe I’m saying this — balance.” Interestingly, the Times applied some context in a follow up article about the breakneck pace of the modern workforce. In it, the Amazon takedown article was referenced with, “some current and former employees complained”. By that point, it was too little too late.

  • Highlighting of hand-picked anecdotes.

The Times mined the interviews for the most controversial and blistering anecdotes and uses those to form the article’s main argument. But a few anecdotes from a narrow dataset are known as outliers, and should be treated as such — i.e. “this happened to one person, which suggests it could be an exception…” But the Times treated these outliers as a foundation for its hypothesis. Which is just wrong. In statistics the process of mining the data for evidence to support your hypothesis is called “data dredging” — it’s considered cheating, since it involves manipulating the data to confirm your hypothesis.

Even fellow journalist George Anders in Forbes called out the Times on this:

“Journalists enjoy the right to be selective, conducting long interviews and then using only short segments in an article. They enjoy the right to interview wide ranges of people and then to build the final story around a small subset. All the same, there’s something worrisome when the discard pile tells a very different story than what makes it into print.

To be clear, the handful of horrifying anecdotes about employees getting edged out because of family funerals, pregnancies, and other personal tragedies were terrifying and have no place at Amazon or any company. Jeff Bezos said as much in his response to the article: “Even if it’s rare or isolated, our tolerance for any such lack of empathy needs to be zero.”

  • Insults and belittling.

Here’s where the Times broader agenda starts to become clear. Several times throughout the piece, the reporters poke fun at Amazon’s core e-commerce business, portraying it as trivial or not worth the hardship required. Here are a few examples: “with words like “mission” used to describe lightning-quick delivery of Cocoa Krispies or selfie sticks”, and “for the cause of delivering swim goggles and rolls of Scotch tape to customers just a little quicker.” I’m really puzzled by these statements.

The trivializing of Amazon’s e-commerce business is just plain strange, and again it raises questions about the Times’ motive. It’s also, again, factually inaccurate. Does the Times think that Amazon isn’t ambitious enough? E-commerce is one of Amazon’s core businesses, and the business that helped it overtake Wal-Mart in market cap, hire over 180,000 people around the world, build the infrastructure (AWS) that supports a large chunk of the internet (including, not surprisingly, the New York Times), build an LA production studio from scratch and within a few years produce a Golden Globe-winning drama series (Transparent), and on and on. The Times doesn’t mention any of these markers of success. Which again, makes the article weaker and raises questions.

I’m writing this not as a rebuttal to the Times story about the culture at Amazon. Nor do I intend to pretend that the company is an easy place to work. The article was correct in general in that Amazon as a challenging place to work. The company demands a lot from employees. The bar for excellence is extremely high. The culture can be rigid and is data-driven. Under-achievers probably don’t last long. But none of that is news. And, these attributes, I would argue, are largely why the company is so successful. They’re also why people like me decided to join the company.

But because the article is so obviously flawed, the Times put itself in the line of fire, and it has me and others questioning its biases and motives, rather than the article itself. As Jeff Jarvis wrote “once one starts to believe The Times might have an agenda, one is left trying to suss out what it might be: against Amazon and its owner, Bezos, who is a competitor.”

Which should make us all wonder: why didn’t the Times more prominently mention that Jeff Bezos owns its main competitor, the Washington Post?