Facepalm: April 16, 2017 Snippets
As always, thanks for reading. Want Snippets delivered to your inbox, a whole day earlier? Subscribe here.
In Snippets over the past few months, we’ve talked a bit about the idea of how complex systems operate: from system encroachment and safety (following the AWS outage) to the role of humans as loosely-coordinated system operators last week. This week we have a story to share via James Hamilton, a VP at AWS in charge of infrastructure efficiency and stability, who also writes an excellent blog on data center design, operation & safety. A few weeks ago, he shared with us an illuminating story about hazard management in a complex environment: namely, the data center managing a major US airline’s operations. (No word on whether it’s the same airline that has been in the news a lot this week for, let’s just say, reasons.)
At scale, rare events aren’t rare | James Hamilton
His account is worth reading in full, but the basics of what happened are as follows: data centers (including that in question) are furnished with switches and backup generators that kick in whenever utility power either goes down or veers outside of acceptable quality parameters. This technology protects the servers, networking gear, and other equipment from surges, power outages, and other bad things; it works successfully almost 100% of the time. The crux of this story is that all of this protection equipment is itself expensive, running in the millions of dollars; so the airline implemented auto-shutoff safety provisions designed to protect the safety equipment from being damaged in turn. Do we see the problem here? By protecting the protection-equipment, you’ve partially undone the safety that you’d so painstakingly created! Murphy’s Law strikes again (and again, and again):
How complex systems fail | Richard Cook, Cognitive Technologies Laboratory
As Cook put it perfectly in his foundational paper: “The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes may actually create opportunities for new, low frequency but high consequence failures. When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures. Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology. These new forms of failure are difficult to see before the fact; attention is paid mostly to the putative beneficial characteristics of the changes. Because these new, high consequence accidents occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.”
We see this principle quite clearly in Hamilton’s example, whereby technology and process created to prevent damage to a $1M power generator introduced a path to $100M failure for the airline. (Incidentally, this was similar to the system failure that led to a power outage at the Super Bowl a few years back.) Two kinds of pressure were responsible: technological (the push to upgrade systems to newer technology with a lower local failure rate) and financial (the understandable desire to shield million dollar capital equipment from blowing up!). When we recognize these pressures, we see them everywhere, particularly around two long-term trends in tech. The first is that modern software is rapidly expanding into complex, hazardous industries: power generation, health care, financial services, transportation, and more. The second is that many of the old technological back-ends underpinning the systems that do work, and which have been held together by sweat and duct tape for decades, are in danger of becoming so old that no one knows how to maintain them anymore:
Banks scramble to fix old COBOL systems as IT ‘cowboys’ ride into the sunset | Anna Irrera, Reuters
In other words, we shouldn’t be surprised to see newer, weirder, and more unexpected forms of failure begin to creep into systems that up until now were at least predictable with their problems. This is a significant opportunity for anyone who actually understands this stuff; it wouldn’t be surprising if one or more of the major important software companies being created today that end up as decacorns were the ones explicitly focused on the problem of safety and “system jiu-jitsu”. Still, it’s also worth appreciating that some of the most facepalm-worthy system failures don’t necessarily come from complex, nuanced forces. Many still come from good, old fashioned poor decision making. As Rick from the comments section in Hamilton’s article tells us amusingly: “I still remember how a major data center in Colorado Springs had redundant power lines into the prem, but both of the power leads looped around into a parallel structure 30 feet outside the prem, and the backhoe got them both about 10 feet later.” Nice.
People who are world-class at what they do:
Instagram finds focus under COO Marne Levine, its ‘Efficiency Guru’ | Deepa Seetharaman, WSJ
Margaret Atwood: the “buoyant doomsdayer” | Rebecca Mead, The New Yorker
Robert Taylor, innovator who shaped modern computing, dies at 85 | John Markoff, NYT
From around the VC community:
Niche networks: rethinking scale for social networks in the era of app inundation | M.G. Siegler
Monte Carlo simulation of sales pipeline projected yield | Andrew Parker
Why great executives avoid shiny objects | Mark Suster
The ongoing AI conversation:
Day One:
2017 annual letter to Amazon shareholders | Jeff Bezos
Amazon embracing profits even as investment grows | Eugene Kim, The Information
Other reading from around the Internet:
Who can name the bigger number? | Scott Aaronson
British Airways and the problem of fees, frills and perspective | Patrick Whyte, Skift
Katie Mack, astrophysicist and Twitter expert | Sarah Scoles, Motherboard
Instgram is eating… everything? | Tara Mann
Synergy in Ethereum: this is just the beginning | CodeTract
And just for fun:
In this week’s news and notes from the Social Capital family, two companies focused around the biggest purchases we make in our lives — and the debt we take on to make it possible.
First of all, we’re delighted to announce that Social Capital has led a fundraising round for Approved. Led by co-founder and CEO Andy Taylor, Approved gives home mortgage lenders and brokers a 21st century software solution for all of their paper-pushing headaches, re-arranging the mortgage workflow and freeing up hours of time for all involved. For something as significant as purchasing a home, an ounce of clarity is a pound of reassurance — not just for homebuyers, but for everyone in the transaction.
Landing a rocket ship in the stone age | Andy Taylor, Approved
Approved announces funding and public release | Andy Taylor
In their six-month pilot, customers originated over $200M of loans using Approved, with some great preliminary outcomes: borrowers were 15% more likely to work with lenders when Approved was available; the paperwork got done in half the time; and loan officers were twice as productive as they were before. That’s a pretty significant win for everybody. We’re excited to help Approved on their journey, and look forward to a lot of time and transparency liberated for every mortgage buyer and seller from here on out.
What other purchase comes close to buying a home? For many, it’s paying for higher education. With millions of smart and educated Americans shouldering a heavy student debt burden, some forward-thinking employers have started to think about student loan repayment as a benefit offered by companies, just like health insurance. Today only four percent of American businesses offer student loan repayment as a workplace benefit, but that figure is expected to rise substantially, up to nearly one in four, within a few years. As David Klein, CEO of Commonbond, tells us first-hand in a recent op-ed in The Hill:
“It’s clear that employers, and our economy, benefit from a highly-educated workforce. But with seven out of ten members in the Class of 2017 graduating college in the red, it might just be high time for employers to get involved in the path to pay off student debt. CommonBond was one of the first companies to help its employees pay off their student debt faster, and we’ve seen the impact that this benefit has on our team’s well-being and overall retention. It’s also served as a compelling recruiting tool for talent.”
Employers and Congress have an opportunity to reduce student debt | David Klein, The Hill op-ed
Employer Participation in Student Loan Assistance Act (H.R. 795)
David’s op-ed comes in support of a bipartisan bill moving through congress, H.R. 795, which helps businesses and employees by giving loan repayment-as-a-benefit the same tax treatment as employer tuition reimbursement. But bill or no bill, the time is now; so CommonBond is currently hiring for many different career positions in New York City, including system engineers, product people, underwriting analysts, and internship positions. The company is at a real inflection point and the time to join is now, so if you’re interested or know somebody who might be, please inquire!
Have a great week and make sure you get those tax forms in the mail,
Alex & the team at Social Capital
