Waterfall or Agile? The best methodology for AI and machine learning

The two most widely considered software development models in modern project management are, without any doubt, the Waterfall Methodology and the Agile Methodology.

An overview of the Waterfall model

The Waterfall approach is the way to go in “consolidated” areas of engineering design. In these fields you can assume that progress flows in one direction. In layman terms, once you make up your mind there are no second thoughts. From here the name waterfall.

Software development purists look at the Waterfall methodology as the model to look at for highly structured projects e.g. Operating System design, real-time codecs, scientific software or software for critical environments.

However, this approach can be deleterious for AI and machine learning projects. Its adoption could lead lead to long development cycles and project failures.

Agile development methodology is much more suited to machine learning projects. The table below summarises some differences between the two methodologies. It also emphasises the major reasons why Agile is probably the best development method for data science projects.

A comparison between waterfall and agile in machine learning

Fail slow: only towards the end of the project one knows whether the project reached its key goals

Fail fast, e.g. if model performance is 70% and the minimum valuable performance is 90% the project can be stopped earlier since it is unlikely that the goal will be reached any time soon

Model deployment happens only at the end of the project

Model deployment occurs as soon as an acceptable enough model is ready

References

Scaling Artificial Intelligence Tagged agile, AI, machine learning, project management, software development, software engineering, waterfall

Originally published at https://amethix.com on August 5, 2019.