Bpipe Version 0.9.11

Download: bpipe-0.9.11.tar.gz

Release Notes

• FEATURE: support for required attribute on notification channels to ensure pipeline failure if channel config fails

• FEATURE: experimental support for virtual file extensions / file type mappings , defined using: filetype : [ 'ext.one', 'ext.two', ...]

• FEATURE: agent will wait for commands to finish if it is asked to run multiple in the same directory at the same time

• FEATURE: support for "queue" command : schedules pipeline to run after another one finishes in a directory

• FIX: only match process owned by user when checking for running bpipe process

• FEATURE(experimental): from clause can now be added in pipeline definition to cause inputs to prioritise from a given stage, with branch matching

• FEATURE: update aws sdk to support latest instance types

• FEATURE: EC2 commands now show name of stage via name tag, eg. in aws console

• FEATURE: accept aws creds from environment if not defined in bpipe.config using standard AWS env variables for token / secret

• FEATURE: support direct transfer of files via scp for AWSEC2 executor. If storage config setting is not defined and 'transfer' setting is true then Bpipe will utilise scp to transfer input/output files to EC2 instances

• FEATURE: allow queue to be overridden within pipeline when sending to JMS channel via explicit 'queue' property

• FEATURE: support replyTo as property for replying to JMS messages (rather than JMSReplyTo)

• FEATURE: improved support for archiving: autoarchive and ability to run archive command from outside bpipe dir

• FEATURE: implement support for 'required' and 'unique' as options on gitlab messages : ensures pipeline reports error status if gitlab calls fail

• FEATURE: support for experimental "when" feature to make stages conditional

• FEATURE: mask 'token' config property as sensitive info from log files

• FEATURE: support for setting R_LIBS_USER with R { r_libs="..." } setting in bpipe.config

• FEATURE: add checks to result message returned in reply-to on JMS messages

• FEATURE: allow specification of gitlab projects by id when sending to gitlab

• FEATURE: show custom title in bpipe startup when configured using title : '...' or title '...'

• FEATURE: support to set env variables for commands via bpipe.config, by added env = : , ... to command config

• FIX: http send not retried on bpipe retry if initial attempt failed

• FIX: errors printed by trying to send JSON via SMTP

• FIX: adjust bpipe agent jvm options to reduce footprint on large servers

• FIX: fix transform of wildcard (eg: *.txt) to multiple outputs resulting in regex error

• FIX: undefined pipeline variable somtimes shows bpipe internal stack trace

• FIX: possible concurrentmodification errors when user-specificed config referenced concurrently in stages / reports

• FIX: fix multilevel branch names not reflected in file names

• FIX: error thrown when creating diagram using 'bpipe diagram ...' Support for reservation of gpus with torque (set gpus=N in command config)

• FIX: body of message relayed in JMS message header (causing parse fail in some clients / protocols)

• FIX: fix regex in transform matching multi inputs only resolving single

• FIX: fix error when literal regex used as transform input specifier

• FIX: previous dirty state carrying over causing unexpected cleanup of good files

• FIX: put retry around output dir watcher to handle race conditions / file system hiccups better

• FIX: Explicitly set pwd for singularity as this seems not always to occur by default

• FIX: fix downstream reference to output created by check failing on retry

• FIX: inability to create gitlab issue with empty content

• FIX: json output double encoded in .bpipe/sent files

• FIX: some commands not running with set -e by default

• FIX: fix warning printed for dirty files even though they don't exist, already notified

• FIX: remove duplicates from printing of test outputs

• FIX: remove printout of shebang stripped statement