High performance Workflow Runner
wr is a workflow runner. You use it to run the commands in your workflow easily,
automatically, reliably, with repeatability, and while making optimal use of
your available computing resources.
wr is implemented as a polling-free in-memory job queue with an on-disk acid
transactional embedded database, written in go.
Its main benefits over other software workflow management systems are its very
low latency and overhead, its high performance at scale, its real-time status
updates with a view on all your workflows on one screen, its permanent
searchable history of all the commands you have ever run, and its “live”
dependencies enabling easy automation of on-going projects.
Furthermore, wr has best-in-class support for OpenStack, providing incredibly
easy deployment and auto-scaling without you having to know anything about
OpenStack. For use in clouds such as AWS, GCP and others, wr also has the
built-in ability to self-deploy to any Kubernetes cluster. And it has built-in
support for mounting S3-like object stores, providing an easy way of running
commands against remote files whilst enjoying high
performance.
Alternatively, build it yourself (see go.mod for the minimum version of go
required):
git clone https://github.com/VertebrateResequencing/wr.gitcd wrmake
The wr executable should now be in $HOME/go/bin.
Complete usage information is available using the -h option to wr and its
sub-commands.
Guided usage, tips, notes and tutorials are available here:
https://workflow-runner.readthedocs.io/