Changes for Bpipe Version 0.9.9.8

Download: bpipe-0.9.9.8.tar.gz

  • Feature: options statement for easy parsing of script specific command line arguments

  • Add workaround to fix possible failure on old pipelines checking file existence

  • BUG: Fix spurious error printed if broken symbolic link present

  • BUG: prevent agent launched commands hanging due to if bpipe prompts for information

  • Feature: add fileSystemSyncTimeMs which allows for customizable delay in file system sync before failing pipeline on missing output

  • BUG: fix regression - transform using regex does not substitute groups

  • eg: transform('(.*)_R1.fastq') to("$1.txt") { ... }

  • Many fixes / optimisations to GCloud usage:

  • make workingDirectory specified for stage automatically placed within mounted storage

  • inherit region from main config inside filesystems, so it does not need to be double specified

  • support preemptible flag passed to google cloud instances

  • add sha1 hash of pipeline path / host to instanceid to make it unique b/w pipelines

  • fix region configured for cloud executor overriding region specified with -L

  • search for root input files in all filesystems, not only those configured as storages for stages

  • fix zone flag missing from gcloud commands, don't die mounting storage if instance not created

  • support using split / partitioned regions in cloud provider storage

  • BUG: Don't start executor pools when user is only testing the pipeline

  • BUG: ensure that executors clean up even if they error in stop command

  • Fix table output format so it shows correctly in GitHub markdown

  • Better behavior for bpipe test with send, and send for missing outputs

  • BUG: input patterns with dir in the pattern causing error / exception

  • Feature: support for reply -to in JMS bpipe agent

  • If the reply-to or JMSReplyTo header is set, the bpipe agent will now send a message containing the original command and pipeline success / fail to specified reply-to address (allows eg: camel inOut routing)

  • BUG: Fix inverted logic causing incorrect detection of file existence when file is out of sync, causing file systems with high latency to fail sporadically

  • Better tolerance of file system sync issues for files referenced using file(...)

  • BUG: from sometimes failing to ensure specified input is resolved in preference to other available inputs preceding it in the pipeline

  • BUG: from sometimes selecting incorrect input in complex scenarios

  • BUG: fix file system watchers still created even if poller watcher specified, hence still resulting in file watcher exhaustion errors

  • BUG: -d option not respected in some situations, incorrect output dir may be used if default output already referenced

  • Feature: Support for Groovy 2.5.6 via optional gradle.properties setting

  • Feature: support for template dirs created by bpipe agent with incrementing values. Directories containing {inc} will be named with auto-incrementing counter.

  • Feature: 'terminateOnError' flag in send attributes causes notification failure as to be treated as error

  • BUG: improve unfriendly / incorrect behavior when duplicate mappings exist in transform

  • BUG: checks sometimes re-executing when not necessary

  • Feature: bpipe touch command updates files from current run of pipeline to new timestamps so they do not re-run on bpipe run.