Cluster Orchestration and still maintain simplicity
Not so long ago cluster orchestration was meant for only few people, gurus. This task forced the business to create another role called devops engineer to focus on infrastructure orchestration.
cluster orchestration issue was solved by having devops engineering to focus on only cluster orchestration. Other guy outsourced to companies which I felt like they were expensive.
As the joke goes you fix one bug you introduce another 5 bugs. There was now disparity between software developers and devops. And as system growed the disparity increased. Deploying and orchestrating code that you didn’t write or you don’t understand was a problem to Devops. In response to this disparity they would define guidelines to be used by software engineers, this then resulted to developers having less freedom which sometimes would impact creativity.
The community tried to close the increasing gap between devops and software developers by developing tools such as chef, puppet, ansible etc. Which really helped to reduce the gap but orchestration of cluster of servers would still be done by few guru engineers.
And finally introduction of docker solved the issue, apart from reducing the gap it made software developers to be devops. To create their app and orchestrate it. Software developer can now develop software and using docker-swarm have a cluster of servers. As Trump had said “Bringing power back to the people”
I have personally tried docker for local development then used docker-swarm with docker for aws (https://docs.docker.com/docker-for-aws/) I had a cluster of 100 machines in less than 100.