Top Developer Skills: Clouds Are Here
The Game has changed — modern Cloud is more like a Planetary Computer, not just a bare virtual machines with infrastructure.
No breaking news here, but quantity had finally transformed into quality.
Microservices are trending. Resources are abstracted away from physical or virtual machines. Architectures are going to be responsive, resilient and elastic.
Cloud Native Computing popular in the community. Many consider “clouds” as an essential part of modern Systems Design.
Trends are tectonic. Get ready to take advantage of upcoming opportunities.
It’s time to add “cloud” skills to your basic skill-set, in addition to Algorithms and Data Structures.
Keep up with the Joneses.
Cloud Landscape
You may feel ok with clouds, but solid understanding is far better. No need to become “all clouds expert”, one modern cloud is enough.
As for me, the Google Cloud Platform is the best choice.
Why? Because GCP is the same place, where Google runs its own products at a global scale.
Here you can touch the genuine implementations of famous Google Whitepapers — the origins of great modern technologies, especially in a BigData area. Hot fresh stuff like Kubernetes, Tensorflow and Apache Beam were born here.
Architecting with Google Cloud Platform Specialization at Coursera is a good place to start.
Best practices and important parts of the ecosystem are shown. Lab GCP account and detailed descriptions of what to do — you may just copy-and-paste all instructions step-by-step. End to end Solution Design during the final lesson.
Good enough to explore the landscape of modern clouds.
Cloud Discoveries
Discoveries depend on your background, so I’ll show you my ones.
Geography is represented with a hierarchy of Zones → Regions → Locations.
Networking is completely global & virtual, running on top of the global subsea cable network.
Traffic ingress & egress endpoints are also global, explicitly managed with powerful API-management services.
During lab exercises, it was fun to refresh good old memories of traditional network engineering — nets, subnets, masks, NAT, gateways, routers, ports & protocols.
I was impressed with the VMs Live Migration, pleasantly surprised with money-saving Preemptible VM Instances option.
Infrastructure as Code is out of the box. Terraform provider is also available.
It was a bit hard to dive into details of Identity & Access Management, mainly because of its flexibility and maturity.
Once again convinced of the importance of security tools, encryption and keys rotation routine.
A lot of fun to play with Hadoop & Spark (suggestion to run Hadoop cluster per job sounds fresh). Quite trivial to run familiar LAMP Stack from Marketplace.
Useful to learn about global data storages. Distributed nature has a huge impact on design, and storage solutions are wisely built on top of each other.
Besides the familiar columnar and bucket storages, it was fun to meet global strongly consistent OLTP. Mind-blowing conquest of the CAP-theorem, achieved by using Atomic Clocks & GPS throughout fast infrastructure!
Insanely fast OLAP solution — querying in seconds, regardless of dataset size. It feels like a short-lived touch to the source of unlimited computational power. No magic here — just a piece of Google’s capacity, leased for a few moments far beyond any capacity of your own bare-metal.
Scalable Enterprise Service Bus is also available — fully compatible with Lambda Architecture, not sure for Kappa.
Many ways to run your code in the cloud:
- serve tiny snippets with Cloud Functions
- cloud-native applications aka App Engine
- streaming & batching data processing with Apache Beam at Cloud Dataflow
- containers cluster with Kubernetes Engine
- whatever you want at VMs with Autoscaling
Powerful Machine Learning complex.
Models APIs for Images & Video, Speech-to-Text & Text-to-Speech, Translation, Natural Language Processing, Dialog Conversations.
Even the HR solution.
Rich Ecosystem for Data Science and Machine Learning Engineering.
Jupiter notebooks, solutions for data exploration and reporting.
Heavy training scales-out in the cloud, powered with hardware acceleration if necessary. Orchestration of models in production is also available.
Cool thing — before feed data to Tensorflow, you can use Apache Beam for data preparation and feature engineering. It makes easy to integrate models into existing data processing pipelines.
Cloud Surprises
Surprise #1: Pricing is represented with much more options than I expected.
Services are carefully described in terms of costs: operations and actions, traffic, storage, memory, computing power. Quotas and Limits are also well documented. Pricing Calculator makes it easier to estimate costs.
Curiously enough, pricing at GCP looks more like a technical specification, than a marketing offer.
In some sense, money — is how computing efficiency is measured in cloud.
Surprise #2: Vendor-lock is quite moderate.
It’s quite hard for me to change mindset towards proprietary solutions.
I trust community and prefer full control with open source stack. Even if a proprietary solution looks great, for me it is very important to be sure I can rid of it any time I want.
Main rule: business-logic should be portable, data should be easily exported in popular formats.
Fortunately, the contribution is a part of Google culture.
Many of key GCP components are at open source (Kubernetes, Tensorflow, Apache Beam, etc) or compatible. Some components are impossible to fully opensource (like Spanner or BigQuery) because to run them properly you need to have hardware & infrastructure similar to Google.
To build your own cloud, SysOps experience required, but it’s the only barrier to move your existing business-logic & data to your own space. Fair enough.
Conclusion
Nowadays, many great opportunities are already available with clouds.
It is convenient to build & maintain quite advanced solutions, even without solid SysOps experience. You may even fit into free quota if the architecting skill is strong enough in terms of cost-efficiency.
Even if you’re a small one, you can easily bring “big business stuff” (Big Data and Machine Learning) into your infrastructure.
Add “cloud” skills to your basic skill-set to meet new opportunities.
Keep up with the Joneses!
For additional details, read:
Do not forget AWS and Azure, even if Google Cloud looks great already!
Feel free to unleash a holy war in comments :)
