Member-only story
Notes from Industry
Impact of Computational Complexity on your Product Value
Pick the most valuable algorithm for your product
The importance of computational complexity including both time- and space- complexities are underestimated especially for the algorithms that becomes a part of a product. For general algorithm development for competitions such as offered by Kaggle, data analysis using ML, model development and optimization for proof-of-concept (POC) the computational complexity is of low importance. However, for the algorithms that go into production systems as a product, computational complexity should be taken seriously due to limited resource availability, time constraints for prediction and model building, and most importantly cost incurred from the cloud resources.
The computational complexity plays a major role in model training where tuning and optimization are required compared to prediction where the complexity is usually O(n*p) where n is the number of predictions and p is the number of features in the data set.
Model Building Paradigms:
- Streaming: Whenever you get a new datum we re-build models using all (or limited) the previous data. Possibly most accurate but expensive, time consuming, and delay in prediction. Ideal for certain…