Member-only story
Command Line Interface with sysargv, argparse, docopts, and Typer
4 ways to pass arguments to your Python scripts
To deploy a pipeline, typically there is a main script, or a single point of entry that runs the whole pipeline. For example, in a data science pipeline, the point of entry of the code repository should orchestrate and run the data, feature engineering, modeling, and evaluation pipeline in sequence.
Sometimes, you may need to run different types of pipelines or make ad-hoc tweaks to the pipeline.
Tweaks may include omitting certain parts of the code or running the pipeline with different parameter values. In data science, there could be a training and scoring pipeline or certain runs that require a full or partial refresh of the data.
The trivial solution will be to create multiple main scripts. However, this will result in code duplication and it is hard to maintain the multiple scripts in the long run — given that there can be many combinations of tweaks. A better solution is to have the main script accept arguments, in the form of values or flags, and subsequently run the appropriate type of pipeline via the Command Line Interface (CLI).
