Why Your FinOps Toils Goes Waste
Resource-centric FinOps efforts will only provide short term fixes but a socio-technical analysis of cloud cost management problems will help you improve the value of your cloud consumption. Here is how to look at FinOps from holistic angle.
For sixth year in a row, in the Flexera’s State of Cloud reports, optimizing the existing use of cloud is the top initiative among surveyed organizations. Yet, as mentioned in my previous article on cloud wastes, I could find much written on cloud cost management solely from a resource management perspective.
However, cloud cost management or cloud financial management is a socio-technical ensemble, that requires analysis from a technical as well as a social angle to address it holistically and with a lasting impact.
Hence, in this article, I attempt to do a socio-technical analysis of typical organisational problems underlying cloud cost over-runs and also talk about the services offered by AWS to assist you. This analysis intends to assist you (CXOs, Business/Product Owners, Architects, and Developers), as the stakeholders of cloud cost management, to better align your actions with each other and achieve a balance between both technical and social dimensions.
What is a Socio-Technical System?
Any organisation, or a part of it, is a socio technical system because any organisation employs people with certain skillset, who work to achieve set goals, following laid down processes, using particular technology, operating on a foundational infrastructure, and sharing certain cultural norms.
Thus, any socio-technical system has 3 social dimensions along with 3 technical ones, namely, Infra, Tech, Process, People, Culture and Goals.
After this brief explanation on socio-technical system, let me give you a view of cloud cost management via each of those 6 lenses.
Cloud Cost Management — via social & technical lenses
Infrastructure — the foundational blocks
Infrastructure aspect of socio-technical system is all about the foundational blocks that need to be in place to manage costs in an efficient way. For example, if there is no proper classification of cost corresponding to each department/portfolio then allocating a cost budget for each department would make no sense.
Some typical problems pertaining to infrastructure are
- Lack of AWS account structure aligning to organisational departments
- Lack of tagging strategy that identifies any AWS resource to its cost center
- Finance, Business, Operations and Application teams operating in silos
AWS services offered to address these problems are,
- AWS Control Tower helps you quickly setup a secure, multi-account AWS environment using AWS best practices that aligns with your organisational structure.
- AWS Organisations service offers tag policies feature to help you standardize tags across resources. Tag policies enforce the accepted value for a tag.
- Also, AWS Organisations service provides service control policies feature to restrict the creation of a resource without the appropriate tags.
Organisation aligned AWS account structure and cost-center tagged resources lay the foundation for managing the cloud spend.
Technology — tools to achieve
Without “extra” skills to handle change, specialization will precede extinction
— The Social Design of Technical Systems by Brian Whitworth & Adnan Ahmad
Technology dimension of the system refers to the availability of tools or mechanisms and deficiencies in those tools to monitor the cost and optimise it.
Typical technology problems are lack of (usage of) tools to
- explore and monitor cost
- budget and report cost
- show plans to optimise rates and opportunities to optimise usage
- identify and eliminate resource waste
- identify cost efficient alternatives
Most of the services offered by AWS will come under this dimension as they serve as a tool to achieve your cost objective.
- AWS Cost Explorer serves as first step in knowing about the spend across resources. You can use AWS Cost Explorer Resource Optimization to get a report of EC2 instances that are either idle or have a low utilization.
- AWS Budgets allows you to setup custom budgets to track and alert if you exceed your budget.
- AWS Operations conductor helps you to automatically resize the EC2 instances based on the recommendations from Cost Explorer
- Reserved Instances, Spot instances, Savings Plan are the options offered by AWS to optimise rates based on usage pattern.
- AWS Application Cost Profiler allows you to split costs between tenants of a shared resource based on consumption.
Process — setup to engage
“Let the process match its objective — democratic results need democratic means.” — The Social Design of Technical Systems
Process aspect focuses on the processes and procedures that are in place to routinely engage people to track their cloud spend and act on it.
Typical process problems with respect to cloud cost management are lack of process to
- identify resource waste and find optimisation opportunity
- report cloud-spend information to all stakeholders
- measure quality of cloud consumption
- collaborate IT Team, Business and Finance
AWS offers services that automates few of the processes, like
- AWS Billing Conductor offers you to customize your billing data to match your desired business structure.
- AWS allows you to configure Billing Alerts via billing console.
- AWS Cost Anomaly Detection allows you to configure cost monitors and define when and how to receive alerts. It leverages advanced ML to identify anomalous spend and root causes, so you can quickly take action. It supports even percentage-based thresholds instead of a fixed-dollar threshold.
- Amazon Data Lifecycle Manager helps to automate the creation of snapshots of unused/under-used EBS volumes (which can be deleted later) to reduce costs
- Amazon S3 Intelligent-Tiering service offers S3 storage costs optimisation by automatically moving data to most cost-effective access tier.
- Instance Scheduler solution on AWS helps you reduce costs by scheduling the start and stop of EC2 and RDS instances.
People — skills and collaboration
Let those with the problem change the system, not absent managers. — The Social Design of Technical Systems
People aspect focuses on the skillset and collaboration among the people on Cloud and its cost management offerings to manage and improve the value of cloud consumption. For example, knowledge of Graviton2 instances offered by AWS can help in choosing those instances for your RDS DB (open source) instances, enabling a 52% price-performance improvement depending on the DB engine, version and workload.
Typical people related problems in the context of cloud cost management are, lack of
- knowledge/training on Cloud (Savings plan, cost efficient instance types, etc.,)
- FinOps (Cloud Cost management) skill — a CoE team
- operating rhythm between application, operations, business and finance teams
AWS offers services that enables upskilling the architects, developers and leaders on cloud and its cost management.
- AWS provides Well-Architected Framework to help you understand the pros and cons of decisions you make while building systems on AWS. It has six pillars, of which Cost optimisation pillar provides an overview of design principles, best practices, and questions along with prescriptive guidance.
- AWS offers Trusted Advisor service that evaluates your account and recommends ways to optimize your AWS infrastructure, improve security and performance, reduce costs, and monitor service quotas.
- AWS Educate is training platform that offers a self-paced training and resources for any individual (beginners), regardless of their education, technical experience, or career journey.
- AWS also offers skill builder platform for anyone with free/paid subscription and for employees from AWS’ partner companies (AWS Partner Network).
Culture — ways of working
Give information first to those it affects. — The Social Design of Technical Systems
Culture refers to sharing of information among and within teams, sharing the responsibilities, handling failures and collaborating across teams.
In this context, sharing cloud spend information, sharing cost-savings responsibilities, handling failures of cost-optimisation attempts, and rewarding collaboration between finance, business, application and operations teams will play a big role in managing cloud spend.
Cultural problems are,
- Tolerance of poor standards — Lack of standards/compliance checks
- Lack of Transparency — cloud cost information not cascaded to all levels
- Culture of top-down managerial concern —cost is not part of application level non-functional requirement
To improve culture of your organisation for better management of cloud spend, you can employ the following ways,
- Customised granular bills (that can be created via AWS Billing Conductor) need to be sent to respective business units regularly. Thus, increasing the cost visibility to all teams so as to act on it.
- Set up Centre of Excellence team for Cloud Cost Management to establish standards and also assist application and operations teams to achieve those standards.
- Introduce cost fitness as a non-functional requirement in application architecture.
Goals — targets to achieve
Give employees clear goals but let them decide how to achieve them. — The Social Design of Technical Systems
Goals aspect of the system refers to the definition and design of necessary targets for the team to pursue so as to manage costs and improve the cloud consumption value. For example, architectural cost targets, business portfolio level cost budgets, and compliance dashboards.
Typical problems seen around this dimension are
- lack of focus on cost related targets
- organisation standards (such as cost budget for an application) derived without the participation of respective teams.
- failure to separate what-is-necessary from what-is-desirable
- no consideration on quality of cloud consumption
AWS offers guidance and ways to create goals that are relevant.
- Budgets need to be defined for each application in consultation with respective teams and should be optimised from there on. Budgets can be defined on cost, utilisation, RI utilisation/coverage and Savings plan utilisation/ coverage.
- Dashboard of cost metrics can shorten the feedback loop for application teams to work on cost as a non-functional requirement. AWS Cost Explorer and the AWS Cost & Usage Report are updated at least once per day. Amazon Athena and Amazon CloudWatch metric streams along with AWS Cost and Usage Report can be employed to build such a dashboard.
- Quality of cloud consumption can be measured only if unit metrics are calculated. For example, for a social media site, cloud spend per engaged user is a unit metric. This metric helps in understanding whether the Cloud spend is in proportion to the growth of user traffic. Unit metrics can be business oriented (such as $/journey) or engineering oriented (such as $/GB stored). While business-oriented unit metrics provide a top-down view for financial reporting, engineering-oriented unit metrics helps understand cost spikes and better forecast costs. Check out this article from AWS for more on unit metric.
Apart from AWS-native services for cloud cost management, many third-party tools are available in the market to compliment them which you can leverage if need be.
“Numbers, the sizes of flows, are dead last on my list of powerful interventions. Diddling with the details, arranging the deck chairs on the Titanic” — Donella Meadows, Thinking in Systems
With this, you would have gotten a fair idea on what kind of problems do occur in an organisation while pursuing to manage cloud spend. Now, “Where do I start?”, “What should be prioritised?” and “Where to intervene in this socio-technical system?” are the questions that you might have in your mind. To find answers to these questions, let’s turn towards Systems Thinking in the next article.
Lastly, in the comments section, tell me about what other problems that you see or other AWS services that can assist cost management. And leave a few claps if you find this article useful.
