The Most Innovative Thing I Did Was a Failure

The Lesson: Sales Alignment Will Eat Product Innovation for Breakfast

Here’s how I learned that lesson the hard way. Absorbing this lesson helped pave the way for the success of EKS at AWS and I’m putting that learning to work at Salesforce for the Heroku business. This, though, is not a piece about Heroku or Kubernetes — I did that already on the official Heroku blog.

I wrote “The Most Innovative Thing I Did Was a Failure” in 2017 about this formative experience. At the time I felt it was too close to the events to publish, but now I think enough time has gone by. I also ran this past a few of the folks involved, and they supported publishing it, notably Errol Olivier and Greg Baltzer. My thanks for your support then and now, and the advice you are always willing to offer.

The Most Innovative Thing I Did Was a Failure

By late 2011 it seemed to me that HP had lost the will to compete in the public cloud. Our morale fueled by the potential of a massive scale out, I led the engineering team through a crash course in open source as we pivoted from proprietary software to OpenStack. The team was exhausted, and despite the successful pivot and launch, HP wasn’t prepared to invest. We were on our fourth CEO in under two years, the Palm acquisition needed billions, and cash was further sapped by the Autonomy acquisition.

Going big was off the table. It was time to find something else to spend my energy on.

I had to come to HP via the Melodeo acquisition and had the confidence of Melodeo’s investors who were now also in the private equity business. This led to my role as Chief Innovation Officer for MTN Satellite Communications. MTN sold connectivity in the global satellite communications market: aircraft, oil rigs, and ships — big ships.

I had endorsement from investors, the backing of the CEO, and a blank sheet of paper mission to build new products to revive the future of a mature company at risk of being steamrolled by the forces of commoditization in a network-resale market where we had domination but little differentiation.

It was time to dig in and understand the customers in our biggest market: cruise ships.

Large cruise ships have thousands of guests and hundreds of crew. IT support drove some need for bandwidth but the primary end user need was for internet access. The cruise lines sold expensive internet access to both passengers and crew by the minute on slow, expensive connections. Rates of $1/minute were not uncommon. Any slowdowns in the connections were interpreted as intentional to stretch out expensive minutes. The service was only used as an emergency service, as an outrageous luxury (because Facebook was apparently an outrageous luxury!), or by those willing to expense the access to their business for on-vacation critical access.

The customer perspectives came early:

• Even though MTN was a satellite communications company, users didn’t care — they just wanted internet access. The cruise lines didn’t care — they just wanted to sell internet access.
• Users were very unhappy with the price and the performance. Penetration rates were miniscule.
• Crew members planned port visits around how to get off the ship to nearby internet cafes that catered to their need to talk to family.
• Cruise ships spend a lot of time in port — much more time than at sea. Moving a cruise ship can be measured in feet per gallon. Sitting in port is more profitable.

Here was the first and most important customer insight: Customers want internet access, therefore we should think of ourselves as an internet access company, and not as a satellite communications company. This insight was the foundation for the innovations to follow, and I began to assemble a new team in Seattle to make it real. Building a team became the main job for a time. “I” became “we”.

We planned to adapt commodity Wi-Fi gear to get internet access to the ships. We found a cruise line willing to let us use a ship as a test platform. We outfitted it with outdoor Wi-Fi antennas and put access points up at a port.

It sucked.

The ships would move and it would fail. Unlicensed Wi-Fi spectrum was jammed with signals. Fiddling with the antenna pointing and location would only work in specific positions and distances. Mounting multiple antennas just increased the amount of noise we captured from the office buildings lining the ports. Mounting the shore antennas very high gave us better reach, but were so high the ships would fail to connect when nearby.

The nature of the solution was right in front of us. Satellite antennas on a cruise ship are two meters across and have to remain pointed directly at a very narrow point where the specific satellite is in a very crowded sky. The pointing has to remain precise no matter how the ship moves and in heavy seas. What if we could find something that worked like that, but really small? We found a company building a small system for the US military. It was used for ship-to-ship communications at sea. We could use it to point an antenna to a GPS location and lock to the signal.

We adapted our cheap radios to their system. We tried it on our test ship. It was awesome. The narrow beam dish antenna we could now use eliminated shore noise. As the ship moved around it would stay pointed, locked to the shore access point. We were making progress.

But, sometimes the port visits on our test ship still didn’t work well. Cruise ship captains have a lot of autonomy. Sometimes they dock the ships in forwards. Sometimes they back their ships into the same berth. On such huge ships, the single antenna was not enough. There was no single place on the ship where we could mount it and not be blocked by the superstructure during turns. A forward mounted antenna might not work at all at port if the ship was backed in.

We had to have two Wi-Fi dish systems, but this came with more complications. Connections were going to be interrupted as the dishes connected and disconnected. A fast internet connection is great, but not if sessions are frequently dropped. An enterprise system could be engineered to handle this, but our customers were internet consumers. Facetime, Skype and Facebook are a frustrating experience if the network is constantly dropping.

We built an overlay network, decoupling the underlying network fabric that was constantly reconfiguring itself onto different paths. Although the system was getting more complex, we now had a major advantage as we could add multiple access points around the shore in various positions. Now if only we had an intelligent control system that could manage heuristics for the two dishes! We built one. We used commodity hardware and cloud-native approaches to build a small standard server cluster, deployment pipelines, and monitoring. We hosted our new shore-side systems on AWS. We expanded the port coverage of our pilot effort and got a customer to trial an entire season in Alaska, where terrain frequently blocks satellites. Our control system began to work very well, swinging antennas through access points like a monkey swinging through the trees — not letting one vine go before the next was in hand. We filed a pile of patents, most of which were eventually granted. We built out the access network from Alaska to the Caribbean and Mediterranean — a major undertaking.

The technology innovation was well in hand, but the total business innovation was not. Especially If you’ve come up through the CTO ranks like I have, take note, as this can be blind spot — it was mine. Tech innovation is easily squashed if not holistically considered by the rest of the business.

End users were happy, but the cruise lines were worried that the fast speeds meant that the users would be online fewer minutes — and charging was by the minute. The sales team was deeply convinced that we were going to cannibalize the satellite business and that this was a very dangerous program to their livelihood. I had also unknowingly stepped on a landmine: the satellite operations team had before my arrival rejected the Wi-Fi-based operation as unfeasible, and we had proved them wrong. Although critical to the user experience of the mixed Vsat/Wi-Fi network, operations rejected the adoption of the overlay network, and managed customer support calls with finger pointing at the new network.

MTN was rejecting the new thesis.

We still had support from the CEO and board, who understood that companies have to be willing to cannibalize their own business or that someone else will. We still had the per-minute business problem to solve, and if we solved the product pricing problem, would we be able to get the cruise lines to make such a big shift? We had hope.

Two massive industry transformations provided inspiration for the solution. The wireless industry was moving to per-megabytes and away from minutes and all-you-can-eat. Amazon was proving that charging customers for what they actually used aligned incentives between the customers and the vendors in a positive way, and allowing product usage to flex enormously without lock-in led to huge usage and happy users.

This led to our boldest innovation yet, this time a business one: we proposed that we buy the end user business from our cruise partners — completely inverting the model. We would pay the cruise lines instead of them paying us. We would control the pricing and internet speeds, and share back with the cruise line. We would offer them a high minimum revenue guarantee, and bigger share in the upside if we could indeed drive the business higher. And critically — we got to switch models from per-minute to per-megabyte. Users who actually used more paid more. If the network slowed they were charged less per minute and were far happier. The better we made the network the more money we would make.

The sales team was dead set against it but we caught a lucky break. Our most innovative customer threatened to replace us because we never came to them with anything new. They were preparing to take our contract out to bid when they never had before.

We had our chance… we pitched it and they loved it. We rolled it out. Cruise passengers loved it. We changed the life of the crew. We organized a new product operations team. We became obsessed with usage patterns and pricing plans.

But all was not well internally. With the new model the customers no longer ordered a circuit and paid for it. We had to manage where and how bandwidth was allocated, and it was now critical to our business that we do that efficiently. With the decreased price and better speeds, the mix of upstream and downstream traffic shifted dramatically.

Satellite operations were far too manual. To drive efficiency we had to extend our new automation solution to the entire network and refactor the way legacy operations worked. My team was charged with operating new products, but the satellite network was a key component. We had to prove we could operate the new service profitably. Legacy operations had incentive to show otherwise and resented the success of the new network.

The sales team viewed the product success with even more trepidation. It was becoming clear to them that we were building a much more SAAS-like model, and the fat days of multi-year circuit renewals were going to fade toward the importance of a much more direct-to-consumer business.

The path forward for the business was now set, and it was clear to everyone (including the board) that for the new path to be viable the company was going to have to refactor in a dramatic fashion. We had the new products, but sales and operations had to be rebuilt almost from the ground up around a consumer focus and comprehensive automation.

The board declined to take the risk and sold the company instead to a competitor. I consider this outcome a failure of my leadership. Technical and business model innovation are worthless in the absence of ability to execute the total vision. It was my job to not only create those innovations but to guide the CEO and board through accepting them.

I should have set the early expectations (as conditions for accepting the mission) that operations were going to need to change, and taken more control earlier when I had more political capital. Automation of operations would have led to better customer outcomes for old products and new, and it is very high risk to build new innovative products on legacy operations. This is especially true when the new product plan requires high operations efficiencies. When the board needed to make a call, we were not in a position to prove the operations efficiency gains would be made, even though to a practitioner of modern operations approaches the benefits were obvious. It’s very hard to tell a company that their large staff of humans is going to be replaced by machines that will do the job much more efficiently. The disruption risk felt far too high.

I was far too slow to respond to the angst in the legacy sales team, which I should have anticipated as a key blocker. Although in the long run the disruption of the sales team was inevitable, I should have pushed the CEO earlier and more urgently to rework sales comp to support the approach during the transition.

When the board decided to cut all spending and investment to prepare the company for sale, I departed to help Samsung chart their path forward into cloud native. Despite that ending, I have a lot of pride in the innovation team and what we accomplished.

I went out on a cruise during the beta product rollout to meet with the customers and crew, and their response to our new product made all the hard work worthwhile. The Wi-Fi network and technology stack to manage it continue to be developed and used to this day, and the patents filed during that time continued to be granted. The product suite rolled into mega-yachts and oil rigs. There was a ferry between Dover and Calais that didn’t have to use VSAT because the approach we pioneered worked twenty miles across the English Channel. The rollout of the new business model, the new global network and the new technology stack was one of the most gratifying in my career. Given the stories I hear former team members tell from time to time about our disruptive effort, I know they feel the same way.

-Bob Wise

October, 2017