Bpipe Version 0.9.9.5

Download: bpipe-0.9.9.5.tar.gz

Summary

This release has a variety of new features with highlights being more flexible options to limit concurrency based on memory usage, and support for integration with message queuing systems (JMS) to both run pipelines remotely and also to send outbound messages.

A range of important bug fixes are included as well. One minor behavior change may be observed for pipelines using the $threads variable: this variable will now only cause dynamic thread allocation if the procs config for that command is either unset or is set to zero. The old behavior caused dynamic allocation of threads when this value was set to 1, which was counterintuitive and confusing.

Features

  • Bpipe now supports an 'agent' that can run in the background on a system and accept remote instructions to run pipelines on your behalf. For now, Bpipe listens to requests via JMS (activemq for now) to run pipelines, but this will be expanded in the future.

  • Improved support for the 'send' command to:

  • Sending json to JMS/ActiveMQ queues

  • Send (POST) json or other content to HTTP(s) URLs

  • Utilitity function 'asQuotedList' formats lists of inputs as syntax compatible with js,python,groovy scripts

  • Experimental support for "sequential" or power operator

  • Print out the outputs to be created when run in test mode, in addition to the command to run (existing behavior)

  • Support for check review states. Previously, checks were were either passed, failed or overridden. Now when a check fails it enters 'pending review' state, and becomes failed only when manually failed after review.

  • Jobs are now automatically throttled based on memory if a memory limit is set using -m flag or limit { memory="<n>g" } in bpipe.config. For this to activate, you must define a "memory" config parameter for commands. An inline / magic memory variable can now be used to reference the amount of memory specified in bpipe.config for a command. Returns memory in GB. eg: exec "java -Xmx${memory}g SomeCommand"

  • Support for memoryMargin command config so that memory variable returns less memory than allocated to a job (to allow for memory overhead)

  • Add maxFileNameLength config parameter that will cause Bpipe to ensure file names do not exceed this length for file systems supporting limited file name length

  • Support for loading BED files as genomic regions to parallelize over, using built in bed command, eg:

bed('test.bed', allowBreaks:false, padding: 20).split(10) * [ foo ]

  • Add support for config for groovy command

  • Check clauses can now omit the otherwise clause. Such checks are executed after a stage exits.

  • Change: dynamic thread allocation is now invoked when procs=0 OR unspecified, not procs=1

  • Preliminary support for jobs in queued / waiting state to be reallocated if a pre-allocated job becomes available

Minor Changes

  • Support for post-command hook to execute a script after every command

  • Add command start and stop time to output from 'bpipe query'

  • Add start and stop time to display in 'bpipe errors'

  • You can now specify java libraries as list in bpipe.config, in addition to the existing string form

  • Add many more details to output shown by bpipe errors

Fixes

  • Fix possible failure of cleanup to identify files as eligible due to incorrectly thinking they are pre-existing files

  • Fix output.dir set differently in consecutive stages sometimes nesting directories instead of creating at top level

  • Fix bpipe test generating a pipeline result, causing it to show up in bpipe stats and other commands

  • Fix 'bpipe errors' failing if non-xml files in .bpipe/results

  • Fix 'bpipe errors' also showing log file tail

  • Fix ranged values not working in proc configuration

  • Fix dynamic threads not used through if procs unset

  • Fix stack traces having double groovy extension in file name

  • Fix segments not correctly represented in json graph output

  • Fix exception/error if double input extension overlaps input extension overlaps output with sng ext, eg. $input.foo.bar happens to match upstream $output.foo

  • Fix completed event sometimes not sent for failed stages

  • Fix failed checks written to std out at end of run

  • Fix situations where checks are re-executed when not required

  • Support supplying input files explicitly to run in run([...])

  • Support for JMS notifications of pipeline events

  • Fix error displayed by 'bpipe log' if no pipeline has ever been executed in dir

  • Prevent recursive load caused defining a segment inside a loaded file

  • Add segment structure to JSON pipeline representation Previously segments were expanded into their constituent stages in the JSON representation, and thus the higher level pipeline structure was lost. Now each node and edge in the group is given a type, and both the segment and its intenral structure are output as nodes / edges.

  • Fix sometimes getting wrong last run pid (thinks still running)

  • Change to make error in loaded script terminate pipeline instead of just printing error and continuing

  • Fix bpipe log stops working after some commands until bpipe run again

  • Avoid printing out cleanup message if zero files to cleanup

  • Move some of the docs out of Language and into Guides

  • Prevent exception in bpipe errors if command state not saved

  • Fix 'bpipe errors' not showing log output if hyphen in branch

  • Fix status command, add regression test

  • Groovy command in script now handles embedded quotes, which previously caused syntax errors

  • Fix error when nested segment made of single stage defined