Bpipe Version 0.9.9.3

Summary

Bpipe 0.9.9.3 contains a large number of bug fixes that affect specific scenarios where unexpected behavior was being encountered. It also includes some major enhancements including the ability to "pause" and resume a pipeline without stopping in-progress jobs, and also support for pre-allocating resources when running pipelines in environments where resources can take time become available after they are requested. The new 'bpipe jobs' command is also worth a look, especially the 'bpipe jobs -watch' option.

Source code: https://github.com/ssadedin/bpipe/releases/tag/0.9.9.3

Highlights

Enhancements

  • Support for preallocated resources for commands and allocated compute resources that persist between pipeline invocations. See the documentation for more details.

  • Support for using Groovy Grab/Grape to add libraries to pipeline (see documentation)

  • Improved output to 'bpipe query'

  • Greatly enhanced, real time display for 'bpipe jobs'

  • More informative display from 'bpipe status'

  • Generic custom options for queuing system jobs (see #193)

  • Improved behavior of stop and status when no pipeline running

  • Support to 'pause' a running pipeline via 'bpipe pause'

  • Tools can be located relative to pipeline for purpose of version probing

  • Experimental support for inline python, sqlite and groovy commands

  • Support to remake specific files via bpipe remake

  • Ability to split genomes into arbitrary sized parts specified in bases

  • Support for syntax of form ${output.withFlag("--someFlag")}

Bugfixes

  • Fix torque executor printing spurious errors when stopping jobs

  • fix spurious error produced by torque module request

  • Fix #209 - torque mem param variable missing in some situations

  • Fix sbatch/qsub failures not recognised correctly, producing misleading errors

  • Fix: files in output dirs not cleaned up if command fails

  • Fix #199: Stop command not halting jobs on some clusters

  • Fix timeout not treated as failure under SLURM

  • Fix split pattern on symbolic links in directory not resolving

  • Fix $outputs.ext (multiple outputs based on extension) broken

  • Fix genomes lacking 'chr' prefix not supported by 'chr' command