The run command

Synopsis

    bpipe run [-h] [-t] [-d <output folder>] [-n <threads>] 
              [-m <memory MB>] [-l <name>=<value>] [-v] [-r] 
              [-u <stage1>[,<stage2>,...]]
               <pipeline file> [<input 1>, <input 2>,...]

Options

-hShow help and exit
-vDisplay verbose / debug logging
-rGenerate HTML report of run in `doc` directory
-dGenerate outputs to folder instead of current directory
-tRun in test mode (see [test](test) command)
-nLimit concurrency to at most `n` simultaneous parallel branches
-mLimit memory usage to specified amount in MB (0.9.8+)
-lSpecify a custom limit (0.9.8+)
-pSpecify a parameter (variable) value

Description

Creates a Bpipe job for the pipeline defined in the specified file and runs it.

The job runs in the background, detached from the current terminal (via nohup), but forwarding output to the terminal.

The -n option limits concurrency that Bpipe itself initiates, however Bpipe will not prevent tasks that it launches from using concurrency themselves. So if your commands themselves are spawning child processes or are multithreaded then you will need to account for that by supplying a smaller number to the -n option if you wish to have an absolute limit on processes or number of cores used.

The -m and -l options add limits that can be controlled by uses blocks that are declared inside pipeline stages. Note that they don't impose any actual constraint on the memory used by tasks that run. They only control concurrency within uses blocks that declare resources.

The -u option causes Bpipe to run until it first encounters the given pipeline stage(s) and then stop. The stages can be specified as a comma separated list. This is useful particularly in development if you want to test a pipeline up to a certain point but don't want it to then go on to run the rest of the commands.

Often it is desirable to make pipelines customizable by exposing variables that the user running the pipeline can set externally. This can be achieved using the -p flag in the form -p <name>=<value>. Multiple -p flags can be provided to specify multiple parameters. Parameters may be read from a file with one value per line by specifying a argument starting with '@' followed by the file name. For example,


bpipe run @params.txt pipeline.groovy

The file params.txt should have one option per line, for example:


-p foo=bar
-p baz=fubar