A to Z: Alphabet’s Cloud Software Contributions

This week is Google Next, Google’s three-day cloud technology event. For years Google has provided leadership around cloud native infrastructure, often leading by example with its own internal solutions that are either open sourced or end up inspiring other projects.

Since Google has managed complex distributed systems and run Linux containers for more than ten years, their internal tools and open source projects provide a good indicator of the challenges businesses face when they become cloud-native at scale. Google’s ability to foretell the needs of the broader community are exemplified by Kubernetes, one of the most successful open source projects, and publicly traded companies built around technology inspired by Google.

In light of Google’s tremendous impact on cloud and open source innovation in our industry, we roundup a list of projects we hear about most frequently: Bazel, Borgmon, BigTable, Dapper, GFS, Go, gRPC, Istio, Kubernetes, MapReduce, Spanner, Spinnaker, TensorFlow, and Vitess.

We know there are more! Please let us know what other projects you are using and what makes you excited about them. We are excited to speak with entrepreneurs who find motivation from Google’s work as their eyes are on the future.

Bazel: Bazel is a build system with advanced local and distributed caching and parallel execution to accelerate builds. It can build and test Java, C++, Android, iOS, Go, amongst other languages. It is used by numerous companies like Asana, Braintree, Etsy, Pinterest, and Redfin and has achieved over 9,500 on Github. Alternatives include Twitter Pants and Gradle, which has seen accelerated adoption over the past year.

Borgmon: Borgmon is the internal monitoring system Google built in tandem with its own cluster scheduling system, Borg. Borgmon helped inspire Soundcloud engineers to build Prometheus, a popular open source time-series monitoring tool that is often used in tandem with Kubernetes. According to a survey published by “The New Stack” last fall, 63% of those surveyed used Prometheus as a monitoring tool in their Kubernetes clusters, significantly higher than second place Heapster that saw 40% penetration (respondents were able to choose multiple tools).

BigTable: In 2004, Google began to develop BigTable, a distributed storage system for managing structured data at a very large size. BigTable is a highly distributed share nothing architecture that supported web indexing, Google Earth, and Google Finance. BigTable inspired HBase and Cassandra, which DataStax built a company around. DataStax is close to unicorn-status with its most recent post-money valuation at ~$967 million. DataStax underscores that Google-influenced solutions can grow into large businesses.

Dapper: Modern applications are constructed from various services developed by different teams and run across multiple separate facilities. Google observed the challenges of identifying performance issues in these disparate environments and built Dapper, a large scale distributed tracing solution that could identify performance issues and provide root-cause analysis. Dapper could handle high throughput, analyzing 2 billion transactions per second. Dapper inspired open tracing, a vendor neutral open standard for distributed tracing that joined CNCF in October 2016, and LightStep, an early-stage startup providing next-generation APM leveraging tracing to diagnose anomalies across all components of today’s software applications. LightStep exemplifies how internal Google solutions have inspired commercial offerings.

Google File System (GFS): GFS is a scalable distributed file system for large distributed data-intensive applications that runs on commodity hardware. Four observations inspired GFS: 1) component failures were becoming the norm so constant monitoring, error detection, fault tolerance, and automatic recovery were integral; 2) multi-GB files were becoming more common so I/O operation and block sizes had to be revisited; 3) most files were mutated by appending new data rather than overwriting existing data; and 4) well-defined semantics for multiple clients that concurrently append to the same file were crucial. Google published the GFS paper in 2003 and inspired HDFS created by Doug Cutting and Mike Cafarella in 2005. Cloudera and Hortonworks went on to commercialize HDFS and are now publicly traded companies with a market cap of ~$2.2B and ~$1.6B, respectively.

Go language: Since Go was released as an open source project on November 10, 2009, we’ve seen the rapid rise of the language and its community of Gophers. Go was built from the ground up and is one of the few languages created when computers had multiple cores. Go is a statically typed language that is compiled to machine code. Programmers note Go’s benefits include its garbage collection, simplicity, and concurrency. Because Go leverages “goroutines” and channels that are more efficient than threads, it works well for writing backend systems like Vitess and Kubernetes.

According to GitHub’s 2017 “Octoverse,” Go is the ninth most popular language, beating out C. The report stated that of the top 10 languages Go was the fastest growing language on GitHub in 2017 with 52% YoY growth. The 2017 Go Survey found that the percentage of developers using Go for work rose to 67% of respondents, up from 62% in the 2016 survey.

Note: Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means there was not enough data for this term.

gRPC: gRPC is a modern open source Remote Procedure Call (RPC) using protocol buffers, a mechanism for serializing structured data. Used internally for 15 years and known as Stubby, gRPC can replace traditional protocols in client/server and web apps. Like traditional RPC systems, gRPC defines a service, specifying the methods that can be called remotely with their parameters and return types. In gRPC a client application can directly call methods on a server application hosted on a different machine as if it was a local object.

gRPC was designed for internal communication between services as compared to REST that was optimized for North-South communication. gRPC can be combined with GraphQL and has support for C++, Java, Python, Go, C#, and Ruby. We’ve seen gRPC used in both container and non-container environments, and it is fundamental to Istio. Probably the most widely known open source project that uses gRPC is CoreOS’s key/value store etcd. Google contributors note they will be focused on improving gRPC’s performance and decreasing its latency in 2018.

Istio: Launched in May 24, 2017, by Google, IBM, and Lyft, Istio is an open-source platform for managing, securing, and monitoring microservices. We’ve previously written about the excitement around service meshes and an in-depth overview of the market can be found here.

Kubernetes: Kubernetes builds on Google’s experience with its own internal cluster scheduling systems, Borg and Omega. Over the past year we’ve seen Kubernetes become the default orchestrator for containers and have heard there are over 30 distributions. We expect the number of distributions to increase over the year as the project continues to gain traction.

We believe the community’s coalescence around Kubernetes allows startups to avoid using resources to build integrations with many orchestrators, so they can focus on building value-added solutions on top. Additionally, standardization allows the best technology solution to win the sales opportunity versus the solution with the specific integration the customer needs.

MapReduce: During the same time as GFS arose MapReduce, a programming model and an associated implementation for processing and generating large data sets. MapReduce inspired Hadoop MapReduce and became part of the Hadoop stack offered by Cloudera and Hortonworks.

TensorFlow: Probably the most well-known Machine Learning (ML) framework, TensorFlow is an open source library for numerical computation and that can work across a variety of hardware platforms from CPUs to TPUs. TensorFlow stemmed from Disbelief, a propriety ML system based on deep learning networks. Originally developed by researchers on the Google Brain team, TensorFlow was released under the Apache 2.0 open source license on November 9, 2015. We’ve written extensively about ML Workflows and published a landscape of solutions, including TensorFlow.

Spanner: Google created BigTable in the early 2000s but found that data with complex, evolving schemas, or that required strong consistency in the presence of wide-area replication weren’t a good fit for the system. In turn, Google build Spanner a scalable, multi-version, globally distributed, and synchronously-replicated database that could be queried using SQL. Inspired by Spanner, ex-Google employees Spencer Kimball, Peter Mattis, and Ben Darnell went on to build Cockroach Labs, which offers an open source cloud-native distributed SQL database, often termed “NewSQL”. CockroachDB has almost reached fourteen thousand stars on GitHub.

Spinnaker: Spinnaker is an open source multi-cloud Continuous Delivery (CD) platform. Martin Fowler defined CD as “a software development discipline where you build software in such a way that the software can be released into production at any time.” While Netflix started Spinnaker, Google joined the project in 2014 and helped launch it to the community in November, 2015. Armory.io offers a version of hosted Spinnaker-as-a-Service.

Vitess: Vitess is a database orchestration system for MySQL that leverages sharding for horizontal scaling. It supports autonomic failover, replication, and rolling upgrades. Vitess runs best in a containerized environment and is cloud agnostic.

Tumblr, HubSpot, Twitter, Facebook, amongst others have implemented and committed to the project. Vitess has recently been voted to be an incubating project in the CNCF. It was the second storage project voted in after Rook. Some members of the Vitess team have gone on to form PlanetScale Data, an early-stage NewSQL start-up based on Vitess.