Cloud Native Organisations

Update 2022: Beware this pattern and creating a distributed monolith through coupling of events. Something i didn’t foresee at the moment I wrote this.

Here I propose a cloud native architecture to facilitate a dynamic culture of innovation and experimentation within any organisation.

It offers the ability to enforce the principles of a “Day One” company in your architecture, an open culture that is focused on results over process (or job title).

This is only one potential architecture to help facilitate this. As you’ll see through the article the architecture matters less than you think. I’m just experimenting. It’s the mentality that matters.

Background

I’ve spent a lot of my time between corporate and startup and one thing that really strikes me is the sheer volume of experimentation, iteration and plain old workload that startups get through relative to corporates.

The customer is all that matters and you die pretty quickly without an increasing number of them. In that pressure cooker, time contracts and a team quickly comes to a “disagree and commit” mentality. They understand that you’re not always going to agree on what you believe is the right approach but you trust in your colleagues decisions, commit to getting the product out the door and letting the customer make the final decision.

Embrace risk

There’s a few reasons corporate organisations begin to stagnate. The primary of which is that large organisations are generally slow decision makers, not necessarily bad decisions makers.

They overthink them, worrying about the consequences of a wrong move rather than recognising that most decisions are reversible and the worry nearly always hurts more than the mistake. Not only in time but also employee morale. After all, the real lessons in life come from failure.

Day One

Amazon labels itself a “Day One” company, referencing the almost frenetic focus on customers that exists in a startup on Day One and which ticks over to Day Two the second you begin to take your customers for granted and switch your focus to securing what you already have through strict process.

Amazon works extremely hard to ensure that a culture of fear never works its way into the company bloodstream. And one of the keys to doing this is differentiating between Type 1 and Type 2 decisions.

Quarterly Quarrelling

Type 1 decisions are not reversible and shouldn’t be taken lightly. They are one way doors that when you walk through there is no going back. Deciding to retire a product. To change your CEO. These are Type 1 decisions for a company and should not be taken lightly.

But most decisions are not and yet we tend to treat them like so. Most are Type 2 reversible decisions, two way doors where if you make a mistake you can simply walk back through the door and start again.

Everyone in your company should feel empowered to make them.

The problem

Most companies are not structured to allow employees to innovate and make these types of decisions on a daily basis however. Some have spent decades working very hard for the exact opposite, to make it it near impossible for all but the most strenuously tested and diluted ideas to see the light of day.

This is especially so in technology where fear of change can reach almost crippling levels. This in turn leads to stagnation, which will be more evident than ever in the cloud era.

Technology companies are growing exponentially, making delays in decisions increasingly irrecoverable from even for companies with a strong cash reserve.

Cloud native adoption is your companies chance to change this. A time to refocus on the customer and return to the principles of a Day One company.

Cloud Native

Cloud Native development is a hard term to define, but most loosely and in an effort to come up with a catchy slogan I‘ll define it as focusing on serving rather than servers.

It’s a way to leverage the cloud to take care of the undifferentiated heavy lifting that generally comes with building technology. You then use that rapid feedback cycle to quickly learn from your mistakes and zero in on the optimal solution.

In a Cloud Native organisation you typically build applications that do one thing well and can then be used as building blocks to construct custom solutions for your customers. Not all that different from lego pieces.

Cloud providers like AWS provide the main building blocks here. Databases. Functions for business logic. Event buses. Machine learning frameworks. Data analytics pipelines.

And by simply leveraging the same mindset AWS had when building these — scalable services that do one thing well — you can tailor the cloud to your organisation and begin to rapidly deliver solutions to your customers.

The real power of a cloud native organisation is the flexibility and speed it can bring to bear on new problems and ideas.

An AWS Reference Architecture

Let’s introduce our architecture for facilitating innovation across your organisation.

(I’m trying to make this general so any reader can grasp the concept so it may seem a bit convoluted).

Let’s imagine we pay for daily reports on mentions of our organisation across all social media. As a global organisation these can be in any language which makes spotting common trends among them a bit of a chore for a single team.

Using AWS we can leverage the Translate service to translate the mentions into English (or any other language), feed that into the Comprehend service (which supports a subset of languages) to identify commonality among them and then catalog and report on it for examination downstream.

So what!?

You’re probably now thinking “That’s all fine and dandy but I don’t see how this changes my company culture”.

Well the key in this architecture is to focus on what we just did by publishing our daily report to the EventBridge service instead of an internal team getting it delivered directly to them. EventBridge allows anyone within your organisation to also listen to this event in real time.

Sure we built a solution that can cater for our immediate needs but the detachment means that anyone in our organisation who has a light bulb moment around social media can now quickly access and iterate on this data.

Imagine your machine learning team has some down time, explores your event registry and realises they could easily tap into these social media mentions to identify which products are mentioned. They do a couple of days spike on the idea and by the end have a newly deployed service which catalogs mentions by product. Let’s update our workflow.

Now your data analytics team realises that between the social media content, the sentiment from the AWS Comprehend service and the distinction between products that they can easily provide a dashboard showing approval rating across social media by product trending by release.

You quickly notice that one of your product features is getting a spike in positive social media mentions since you released a new upgrade and decide to dedicate more resources to it at a company level.

The Flywheel

You can see here that with minimal need for coordination and management three seperate (imaginary!) teams were able to quickly innovate on ideas which led to an increased level of visibility across the organisation into how our customers are talking about us.

This architecture facilitates what people often call a company fly wheel. As more and more services become available and more business events, the potential for innovation and experimentation begins to accelerate.

This brings more business value to your customers which brings in more customers which generates budget for more services in your cloud native ecosystem.

Align this to principles like infrastructure as code, test driven development and continuous delivery and you have a framework for rapid experimentation at low capital and resource cost.

Some key points

• Our ultimate goal here is to enable a cultural shift where those responsible for decisions feel comfortable empowering their employees to make them.
• Teams can feel safe deploying code all the way to production (once aligned to certain principles) to experiment as they will be attaching parallel listeners to events which has no impact on other applications.
• Parallel testing of results is one of the quickest ways to cut through bureaucracy and focus on delivering value.
• Seperate accounts can be created for experimental branches however to ensure a limited blast radius should any issue occur. This is just good practice at the start to ensure it doesn’t affect your mentality of experimentation.
• This accelerates the need for processes like DevSecOps to ensure that you never introduce a security vulnerability or compliance issue.
• Any attempt to do this without aligning to strict DevOps principles like Infrastructure as Code and deploy from source control will end disastrously. Put in the time to establish strong DevOps mindset and principles to facilitate this shift in mentality.
• All principles should be enforced through rules in the system.
• Seek help from professionals before starting your journey. This is not an easy thing to get right but when established it is powerful . You want to feel 100% confident your flywheel is heading in the right direction.

Conclusion

By leveraging a managed event bus at the core of your cloud native operations and reusable micro-services, organisations can rapidly progress on multiple problem domains in parallel while ensuring organisational sharing of information and solutions. A smart “hive“ constantly innovating and serving your customers.

About me

An AWS Certified Solutions Architect Professional with a passion for accelerating organisations through Cloud and DevOps best practices.

If you want to work together contact me over on brianfoody.com, on LinkedIn or Twitter for a chat.

And don’t forget to follow the Lambda Lego publication!