My journey through the cloud
A Firsthand Account of Amazon Web Services Through the Years
At the end of 2009 I left my corporate career at SAP to start a new job as a Software Production and Release Manager at Indicee — a fledgling startup trying to reinvent BI for the cloud (we called it Crystal Reports 2.0 at the time.) The decision couldn’t have been more transformational. At Indicee, I met Amazon Web Services, or AWS for short — it was a love at first sight!
I always liked the idea of online services (or what we know as the cloud today.) I have been an early adopter of many of them and still use some notable ones today: Gmail in 2004 (switched from Yahoo! Mail which I used since 1999), Linkedin in 2006, Picasa in 2007 (Google Photos today), Google Docs in 2010, Github in 2010.
Still, none of them excites me as much as AWS did back in 2010 and still continues to do so today.
The AWS journey to become the largest provider of infrastructure on the globe started with a very modest announcement on Nov 3, 2004. The Amazon Simple Queue Service (SQS) was delivered as a Beta to developers that needed to integrate message queues in their web applications. For the first time, developers didn’t have to deal with the messy and often difficult effort to deploy the infrastructure and configure the software as the integration of the service in the rest of the application stack was very simple and consistent between development, testing and production.
Outside the startup community, the rest of the IT world didn’t take notice until AWS announced the launch of Simple Storage Service (S3) on March 13, 2006. Amazon boldly proclaimed that “Amazon S3 is storage for the Internet.” Boy were they right — there is virtually no web deployment today that doesn’t somehow use S3 directly or indirectly.
Many enterprise IT companies use it too for their backup or file storage needs (often through other services like Dropbox — even though Dropbox eventually moved some of their storage to their own data centres they still use S3). Very shortly after launch, an announcement on July 11, 2006 proclaimed that “Global enterprises like Microsoft are using Amazon S3 to dramatically reduce their storage costs without compromising scale or reliability.” The world started taking notice and the term cloud started entering the IT dictionary.
Then AWS did something no one expected — on August 24, 2006 they launched their Elastic Compute Cloud (EC2) service in Beta. While very limited initially, the EC2 service found raving fans in the developer community. For the very first time in the history of software development, they could get their hands on as many virtual machines (or compute instances as EC2 calls them) as they wanted for their development needs, to run their tests and even deploy their web applications in production.
Gone were the days of asking IT to give you a server and waiting for weeks or often months before they provide you with one that could do the job — if you were lucky that is, as often your request would get punted by other, more important, requests, or you would get a server with less power than what you asked for due to budget limitations or too many concurrent requests.
In only 2 short years, EC2 became a staple for deploying solutions by many startups. The pay as you go model introduced by EC2 was revolutionary. The AWS cloud let them quickly provision servers as they needed them to run their fledgling businesses that were strapped for cash.
In this period the EC2 service got a number of important upgrades: ability to deploy in multiple availability zones — AWS’s core building blocks for deploying fault tolerant and highly available applications that would continue to run even when one of the AWS data centre fails; support for Elastic IPs — ability to assign your own IPs to your servers so you can keep your DNS static even when servers get replaced; ability to deploy in Europe and later around the world; etc.
By the end of 2008, EC2 was capable of running different flavors of Linux, got support for Windows and many vendors (including Red Hat, IBM and Oracle) offered support for their software in the AWS cloud.
In 2009, AWS started innovating with concepts like elastic block storage, spot instances, software defined networking, monitoring, auto scaling, load balancing, managed SQL databases, etc. It suddenly became clear that the platform could provide much more value than traditional managed hosting providers. The cloud let you run not just virtual machines but higher level compute services too.
AWS showed commitment to remove all of the undifferentiated heavy lifting (the muck that comes with hosting operations) and allow businesses to focus on their applications instead. It made the lives of both the developers and IT operations engineers easier. It enabled programmatic access to numerous compute services and ushered in the era of infrastructure as code — where everyone is a developer and the boundaries between the traditional roles of software developers, QA professionals and IT operations started to meld.
When I met AWS for the first time in January 2010, I was mesmerized. My team and I managed to completely automate the testing and production operations in a short few months (while in parallel we moved from svn to git and the dev team almost fully replaced the original technology used in the MVP version of the product with a new one).
Every night, a test environment would spring up out of thin air and our only QA person would come in the morning waiting for fresh full regression test results from our increasing automated test suite. If anything was wrong, the environment was available for troubleshooting so issues could be investigated and fixed before new functionality was brought into the code base. Otherwise, it would dissolve back into the ether, leaving no trace behind beyond the logs in our Continuous Integration (CI) tool — CruiseControl (if you never heard of it, it was the predecessor of Jenkins and similar tools).
Little did we know we were doing DevOps without knowing as the word hasn’t entered our vocabulary or the wider IT consciousness back then. We simply loved automating everything and AWS made it possible. We were drinking the kool-aid and automated ourselves out of our jobs, which felt wonderful!
Some define 2006, i.e. the year S3 was released as the birth of AWS but I prefer 2004 and the launch of SQS. The humble beginning of AWS shows how a simple idea — hosting a service aimed at developers — can revolutionize an entire industry and bring to market technologies that were unthinkable before due to the intensive capital cost and expertise required to consider them.
Today AWS is a behemoth — much of the Internet runs on it and even the largest enterprises in the world in some of the most regulated industries or government agencies are going all-in on AWS. AWS is an undisputed leader in cloud infrastructure and runs much more compute power than all other significant players combined.
Probably more importantly, AWS is an unstoppable innovation machine that uses the basic infrastructure that was developed at the beginning to bring ever more innovative and capable services. From micro-services, big data, to data analytics, IoT, machine learning, AI, mobile and game development, etc.
AWS makes it ever easier to experiment with technology and try new solutions that would be prohibitively costly or outright impossible for most companies to even consider. And it is all only early days — each year shows how much more is possible and how much more demand for an increasing amount of computational and application services can be generated when you let developers dream of more and more applications and explore new ideas with very little friction or cost of failure.
In 2012 — after leaving Indicee for another stint at another startup, Vivonet, where I pushed for adopting AWS — I started consulting other businesses to help on their cloud and DevOps journeys, which lead to co-founding TriNimbus in 2013 as a way to continue doing similar work but at a larger scale. Since then we have helped numerous software companies take on a journey to embrace the cloud and DevOps and increasingly more enterprises are joining on the same path. With TriNimbus, we’re touching many businesses — from fintech companies, to telcos, banks, life sciences organizations, and SaaS-based businesses of many different kinds.
Over the years the focus has been shifting from optimizing cost, designing for security and automating deployments to empowering project teams to self service their infrastructure needs, enforcing infrastructure as code principles and adopting DevOps culture, while automating IT risk monitoring and governance and moving IT from a cost centre to a value-add service provider to their organization.
While AWS enabled me to do this, the most important part of my cloud journey is the people I work with every day. The Solutions Architects and DevOps engineers and everyone else at TriNimbus is very passionate about improving the way technology is developed and runs and love sharing their experience with our customers!
The folks at AWS have trusted me and my business partner, Jarrod Levitan, from the early days of our vision. They encouraged their customers to work with us, helping us along the way to build and grow a new kind of IT consulting and managed services business — one born in the cloud and based on software roots and best practices. Their customers increasingly want to use the power of the platform but their existing IT vendors and managed services providers are not able to follow them so they look to AWS to introduce them to new partners.
The customers we work with are also big promoters and tell their friends about the great work we do and urge them to work with us. Many even bring us into their new companies when switching jobs after working with us in their previous role. We’re committed in creating value for them and they trust us to do our best and partner for the long haul.
One of the benefits of doing what I do is that I have done it long enough to be able to see trends and I am doing it across many businesses and industries that I can see things in the aggregate and identify patterns and opportunities my customers may not be able to see. One such realization is that technology will become important to every single business on the planet — the distinction between technology and non-technology companies will diminish since every company will be a technology company.
I often ask potential customers if they see themselves as a technology company. Many come to us with the goal to “reduce their technology footprint” as they found it too costly to manage and it slowed down their business. After some digging, I invariably find that what they really mean is that they would like to reduce their reliance on old-guard technology (aka virtual machines running old business systems like ERPs, file servers, email servers, etc.) and move to experimenting with new ideas and test new services (from analytics, to IoT, machine learning, AI) without huge upfront cost or irreversible impact from failed experiments.
The old-guard technology has been keeping them too busy dealing with the heavy lifting and muck inherent in deploying and managing it and helped turn IT into the enemy of the business. In most businesses, IT is seen as a money pit — the IT managers keep asking for more money to hire more people to run the existing technology and fail to deliver anything newer, faster, or better. The IT leaders are hungry to change but don’t know how or where to start. They’re looking for partners who can work with them to provide unbiased and transparent advice and help them.
This is why I love doing what I am doing. I wake up every day being excited about the value my business is bringing to the many customers that work with us. I walk in my office feeling blessed I get to work with so many passionate and experienced people and humbled by their desire to help not just our customers but each other and ultimately grow our business together.
Even though I have been journeying in the cloud for a while now, I believe it is still early days and the potential ahead of us is enormous. We’re at the verge of a revolution that will be several magnitudes bigger than the industrial revolution that defined our time so far. The cloud is moving fast and many people want to join the bandwagon but they quickly realize their past experiences make them ill equipped to appropriately utilize what’s on offer. That is the paradox of the cloud — we have only started to scratch the surface but so much has happened since the beginning that people are already left behind and the gap is widening at an increasing speed.
I think we need to pull as many of this people forward by educating them and guiding them on their own journey. As the Amazon leadership team is fond of saying, “there is no compression algorithm for experience.” If you’re starting your journey now, it is less lonely with a partner that is willing to share their experience by working collaboratively with you — you know where to knock! ;-)
