First off, great article. I think airflow might have stemmed/been inspired from the implementation of Apache Oozie and hence there was a design decision to couple orchestration with runtime implementation of the tasks themselves _on the same node_. This has been one of Oozie’s fallacies.
So, this distributed approach was something I hoped _was_ the default, where one would be able to deploy operators on remote nodes.
I’m not sure what timeline this article was written at. Does this gel with https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/ ?(I think it’s part of airflow incubator’s base library now).
Was the result of your
KubernetesJobOperator ever merged into public libraries?