Reflections on Kubeflow configuration, packaging, and deployment
These are very exciting times for the Kubeflow project. We’ve just held our second Contributor Summit in Sunnyvale CA, and the talks and participation were amazing. The SCaLE 17x Linux Expo dedicated an entire Kubeflow Day bringing together both community members and adopters in Pasadena CA. The increase in contributions to the project have been overwhelming and this community works very hard to stay on top of it all.
The best open source projects thrive on change and challenges, and it takes a robust and responsible community to adapt and tackle those changes effectively. One recent change that the community is adapting to is the announcement after the acquisition of Heptio by VMWare that the ksonnet project will be retired.
The original community of the Kubeflow project was very small, just a handful of engineers with finite bandwidth. Besides scaling the community, there was the job of scaling the actual technology too! There were many important choices to make in the goal of starting a new open source project for machine learning on Kubernetes. One of those was tooling for authoring configuration and deployment of the new platform. ksonnet, although a new project itself, was selected at the time based on a very promising DSL feature set: local variables, lambdas, modules, imports, mixins, concatenation and merging, and template parameters. These expressive Kubernetes-native extensions to the Jsonnet language helped Kubeflow early on to integrate a rapidly expanding set of components using a common deployment pattern.
Well over a year since the project was announced, Kubeflow is comprised by a much larger and diverse open source community of engineers, writers, and program managers. These people bring their passion, experience, and expertise to bear on the development of a comprehensive open source machine learning framework that today includes:
- Curated ML notebooks
- Hyperparameter tuning
- Inference serving
- Distributed training for TensorFlow, PyTorch, and MPI
- ML pipeline management
- Model experimentation and visualization
- And so much more
The “how” of Kubeflow configuration of these components of the framework is ultimately secondary to the capabilities of the constituent pieces. Make no mistake: it is still highly important for the Kubeflow project to have consistent standards and tooling for authoring component configuration, packaging, and deployment. Irrespective of other abstractions, Kubernetes ultimately enforces that via resource manifests that are externally represented in YAML or JSON. However, the specific technology selected and adopted for component tooling abstractions can and should be subject to review at any time in the project lifecycle. Painful as that may be given the existing investment in source code, that doesn’t mean it shouldn’t happen. The ksonnet announcement has simply afforded the project an albeit unexpected inflection point at which we can as a community take stock and evaluate other choices. And today, there are quite a few: Helm, kustomize, Kapitan, Pulumi, etc.
We have already begun the evaluation of some of these options. kustomize is a very intriguing choice and a proposal for its adoption has been shared in the community. There has been some exploratory work already committed to the code base that makes use of kustomize for the new notebook and profile controllers. Also, many proponents of Helm have chimed in on the community mail list. The initial target for adopting a new strategy and tooling is 0.6. Finally, although ksonnet development has stopped, its final release is still available on GitHub and provides all the features necessary that we require for now to sustain development of Kubeflow in the interim.
So there you have it. Life in the fast lane. If any of this sounds interesting to you then please consider joining our community. Help us solve these and other exciting challenges!
https://www.kubeflow.org
https://github.com/kubeflow
https://twitter.com/kubeflow
