My personal journey to the cloud from my first job on a trading floor to startups
“Hi, my name is James. I’m a self-confessed cloud-oholic and I’ve been off-premises for 7 years now.”
There’s no self-help group for someone like me — technology is in my blood. I owned my first computer when I was 5, and from then onward I collected technology like most kids had stuffed toys.
I started writing for a local computer magazine at 12, then a national UK magazine. When I got to college, I had a regular column on a British platform called Oracle Teletext.
It would have been easier to do cool kid things and hang out at the mall but spending every waking hour thinking about all the problems that can be fixed with machines kept me too busy.
My first job on a trading floor
After earning a CS degree at college, starting work as a software developer was a jarring experience. Having to work in teams of developers is not something they taught at school, nor did they train anyone on how to make sense of ancient convoluted systems. At school, you start from a blank screen and build something beautiful all by yourself, which very rarely happens at work.
My group built electronic trading software for hedge funds, portfolio managers, and the like. Most trading happens at the open and close like a twice-daily Black Friday that stresses the back-end. Our infrastructure was killing us as the platform grew and the amount of data choked any attempt to solve the problem without taking the system offline.
Best of all, our users were traders — and they were assholes on a good day. Their needs changed constantly and were communicated telepathically. They completely lost it during an outage and had no tolerance for bad data, missing functions or sloppy UI.
Smart, volatile and demanding users — working for them ended up being the very best training for today’s user base, and it taught me several life lessons:
- User don’t know what they want. At least, not the final version of what they want. They can read the road for the next hundred feet but anything beyond the headlights is unknown. The paradox is we can only know what to do build in the short term, yet our scaffolding has to work forever.
- Developers underestimate how much effort is involved. The one constant across all developers is a total failure to estimate to how long something will take. 80% completion happens quickly and then the last 20% takes forever, if at all. You pad their estimates, your boss pads your estimates and the delivery is still late.
- Stability is everything. Your system should not fail. Do not, under any circumstances, allow it to fail. But if you do fail, every single outage must be investigated and remedied so it never happens again.
On Error Resume Next: Product Management
In the mid-2000s, I moved to the Bay Area and worked in startups for several years as a Technical Product Manager. This was during the phase when everybody who previously had an idea for a website now had ideas for mobile.
In the Bay Area, a Product Manager is a coder who is yelled at by customers and also produces road-maps nobody uses.
As we moved into mobile, our users were regular people with cell phones, and our competitors were either well-funded startups or the established technical luminaries. Our development teams were much smaller, budgets were tighter and yet our epic aspirations didn’t seem to notice we were horribly equipped for success.
Mobile made scaling problems insurmountable for start-ups — buying new servers sucked up budgets, configuring load balancers and database replication wasted development time that should have been spent perfecting the UI. And investors and founders, usually bored with the the grind of their real jobs and attracted to the gold rush, were on a mission to become the next billion dollar app with no revenue and an army of users.
At the time, there was iOS, Android, Windows and BlackBerry, all using different frameworks and languages, and it looked like these could fragment further. We were trying to put together apps that are essentially a dozen screens which could have been built as a .NET desktop app in a day. And yet we did manage to release apps, solve problems and build some businesses.
I learned:
- You don’t know enough. Your team’s knowledge has gaps in networking, security, scaling, electrical engineering, machine code, you name it. When you face problems that veer into these areas, it’s like quicksand for your product. Developers like tough problems and have curious minds, so these types of issues are a siren’s call.
- Complexity is death to progress. When your team owns all the pieces, they write complex code that locks systems together. But when developers can only use APIs to talk to other systems and don’t know how they are implemented, they write simple code that makes the system modular.
- Dreams aren’t code. If you can’t make your idea function in a spreadsheet or a flowchart, it cannot be built in code, no matter how simple the investor or VC says it is.
Discovering a better way
Sometime around 2010 it became clear to me that as a development group, we could confidently write solid applications running on machines in the same building. But deployment was difficult — and once apps hit production they weren’t performing as well.
We had been using some cloud apps for a while but hadn’t seriously used AWS until it became absolutely necessary. A client app had started to gain momentum and we didn’t have the money to scale up on-premise, so we became AWS users very quickly. It was a fortuitous but mildly alarming moment to realize we didn’t have any alternatives — but it quickly became the de facto way to build our products.
I had some lightbulb moments during this time:
- Infrastructure is hell. It brings out the inner tinkerer in everyone, and it’s a distraction that stops you writing code. You also can’t manage it well no matter how hard you try. So don’t.
- Dev-staging-prod doesn’t work. It’s not sophisticated enough, doesn’t stop bugs reaching the customer and ultimately just provides an illusion of quality. Every service needs versioning at every stage with incoming traffic routed accordingly.
- Agile is beautiful. We were doing it while also doing waterfall because that was considered professional. When I read the Agile Manifesto I almost wept — I knew this was how we would build software from now on.
What happens in Vegas … becomes a career
In 2012 I attended the very first AWS re:Invent conference in Vegas and that changed everything. Witnessing the entire ecosystem around the platform, it was obvious that many people had been grappling with the same issues and there were a slew of great solutions available.
There was a haunting question about why nobody else was offering this — Amazon was the only game in town and either they were incredibly prescient or we were all being gleefully over-optimistic about this whole cloud thing. This lag continued for years — it gave AWS a 6-year lead over its competition which is why its capabilities still smoke the competition.
In our shop we weren’t the first to the cloud by any measure but we embraced it wholeheartedly. Within 6 months there were a number of unexpected side-effects:
- We became truly agile. Our users still didn’t know what they wanted and the devs still underestimated the work, but the dynamic in building products had changed. We could spin on a dime and make radical shifts without blowing the house down — or blowing the budget up.
- The things we didn’t understand well were understood for us. Cloud took many of the computer sci-ency problems away and solved them. This allowed us to focus on building only the apps and our productivity (and profitability) sky-rocketed.
- Our apps became really good. Many weren’t popular and didn’t survive investment rounds but they were extremely stable, scalable and looked like the products of a much bigger team. I cried for the apps that didn’t make it.
My future as a Technical Product Manager in cloud
In using cloud solutions as the backbone to all the products I’ve worked on, I’ve had to step up my technical game constantly. It’s not enough to be a Product Manager with road-maps and wire-frames — I need to know reliable patterns and trusted practices to create the best technical architecture.
This has meant constant training, taking on programming projects and learning new frameworks as the environment changes. It’s also meant making a commitment to conferences and workshops, which has become an automatic line-item in my budget.
On the business side, cloud has given me the confidence to assess viability and likely cost, predict timeframes more reliably and help business partners understand where the business ideas and the technology meet. In many ways, the concepts between agile, cloud and lean are so intertwined that I often think they are different views around the same thing.
Fail fast, waste little, learn constantly and always deliver customer value — cloud is central to making this work.
There are still a few road bumps
There are still plenty of naysayers. I worked for some more traditional companies after the California days and it was like jumping in the DeLorean and setting the clock to ‘Fail’.
They all grappled with an aging, fragile, expensive IT infrastructure that delivered limited business value and had no hope of helping them innovate or differentiate in the future. Those companies are waiting for a generation of executives to retire and competitive threats to reawaken the appetite that once made them giants.
There are also the fakers in the industry, the ones who for years dismissed cloud, laughed at Amazon and claimed it could never work. Now they scramble to promote their own clouds with the same limited tools and restrictive contracts they had on-premise.
The me-too players like Oracle serve to bring the laggards into the cloud ecosystem but they offer nothing fundamental or game-changing to the technology. 5 years ago they said cloud wasn’t secure and now they say only their clouds are safe, so I suppose fear can drive sales in anything.
But I live by mantra “Go where you are celebrated, not tolerated.” I’m not here to convince yesteryear’s IT professionals that our industry’s change is accelerating geometrically. I’m here because I’m committed to using the cloud and its toolbox to build the next generation of software that solves the next round of problems. I want to get to machine learning and AI, and move from onClick to onPrediction — the cloud is where all of this will happen.
So that’s my story. Most of us geeky kids who grew up with computers didn’t become Steve Jobs or Jeff Bezos but it’s been an amazing ride. The opportunities are everywhere and the future has never been brighter. My name is James. I’ve been a self-confessed cloud-oholic for the last 7 years. I don’t think that’s ever going to change.
