Rosetta pipeline: A showcase and comparison of pipeline frameworks

A lot of pipeline frameworks
exist, each containing
a different (and sometimes exotic) syntax. Because of this, it can be
confusing to choose the appropriate framework for a particular use case.
This is further complicated by an inconsistent terminology between
frameworks. For example, rules (make and snakemake), processes
(nextflow), tasks (WDL), jobs (CWL) and stages (martian) all refer to
the exact same concept in different pipeline specifications.

Here we try to create:

• Example workflows for each framework with some common workflow
  tasks. They showcase how each framework is used, and where
  the similarities and differences lie. They are also useful to learn
  a particular framework by example. This is inspired by Rosetta
  code.
• A working document on the similarities and differences between
  pipeline frameworks. This document introduces a
  consistent terminology of features that a pipeline framework can
  have, while listing alternative terms along the way. The discussion
  is meant to be neutral and objective, in the sense that it simply
  lists possible features of pipeline frameworks, along with some
  advantages and disadvantages of having that particular feature.
• A comparison of the features of each framework,
  structured in the same way as the working document.

Structure

• tasks contains the task description and examples for each
  framework.
• comparison contains the working
  document and a comparison of frameworks.
• containers contains the dockerfiles for each
  framework. These are used for executing the example
  workflows.

Tasks & examples

Running the examples

1. Install conda and docker
2. Clone this repo: git clone git@github.com:komparo/rosetta-pipeline.git && cd rosetta-pipeline
3. Install the conda environment: conda env create -f assets/env/environment.yml
4. Activate the environment: conda activate rosettapipeline
5. Run snakemake: snakemake. A first build can take a while because
   all docker containers have to be built. To run only one framework,
   use snakemake --config framework_id=nextflow.

Contributing

We welcome contributions of any kind. See
assets/contributing.md.

A contribution implies that you agree with the Code of
conduct.

Further reading

Opinions

• Sam Minot on how an ideal workflow manager looks
  like
• Experiences when creating an RNA-seq
  pipeline
• Reddit discussion on Toil vs Snakemake vs
  Nextflow

Lists and rankings

• An awesome list of
  frameworks
• Popularity ranking of
  tools

Reviews

• “A review of bioinformatic pipeline frameworks” by Jeremy
  Leipzig
  (https://github.com/leipzig)