You might be interested in the SciLuigi library, which is a thin layer on top of Luigi (you should…
Samuel Lampa

Hi Samuel,

Thanks for reading! I had seen SciLuigi a while ago, and I think there are a lot of cool ideas in there. In fact, I borrowed the SciLuigi idea of using a separate “workflow” task, which is just a WrapperTask that provides a single input to catch all the different parameters, then `yields` the upstream tasks with these parameters set. This was a nice workaround to the nightmare of sending a bunch of parameters all the way up the dependency chain, passing through tasks that don’t actually use them. It’s now possible to set parameters for any upstream task directly from the Luigi command line tool, but for complex NGS pipelines it’s still nice to have the ability to decide on the fly which tasks you want to yield from the requires method of a workflow.

I never fully adopted SciLuigi, though, because even though I like the idea of flow-based programming, I was getting a lot of mileage out of building base task classes, with only run() and output() defined, and subclassing this with a custom requires() method for a specific workflow. If you combine this with returning dicts as Task outputs, while trying to standardize key names as much as possible, you can get pretty flexible with rewiring your workflows.

I do agree that SciLuigi and flow-based programming is an elegant way of dealing with branching and rewiring, though, and if the complexity of our pipelines starts getting out of hand, maybe I’ll have to revisit your library.


One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.