v0.10.0 (2019-03-22):
Fixed a bug that raised exception with empty commands
Fixed the pipeline profiling issues
Major updates to internal systems: Switch to attmap
Revamped way of handling child subprocesses which should lead to more efficient memory monitoring of piped subprocesses, and more consistent handling of rogues subprocesses during pipeline failure.
Added force mode to ngstk gzip and pigz use.
Changed documentation from sphinx to mkdocs.
Fixed a bug with python3 output buffering
Implement multi-target commands
Fixed a bug that had prevented new start mode from working in certain cases.
Allow user to change units of memory passed in with default pypiper cli.
v0.9.4 (2019-01-31):
v0.9.3 (2019-01-31):
v0.9.2 (2019-01-30):
v0.9.1 (2019-01-29):
wc returns leading whitespace, which caused these functions
to fail.v0.9.0 (2018-11-19):
Use psutil to track aggregate memory usage for processes that spawn
children. This results in accurate memory records for these processes.
Individual commands in a string of commands connected by shell pipes are
now treated as individual commands, and and monitored individually for
time and memory, and if a single component, fails, the entire string will
fail. Previously, only the final return command was recorded, as in bash.
Various other small improvements (like waiting checking for dynamic recover flags)
v0.8.1 (2018-09-20):
Fixed a bug that caused a problem for some pipelines adding groups of pypiper args.
Improved the run waiting method to immediately stop upon job
completion, rather than minute-increment polling. This should improve
performance particularly in pipelines with many, medium-runtime steps, and
improve accuracy of timing profiles.
v0.8.0 (2018-06-15):
Implemented 'new start' mode.
Improved error messages and exception handling for missing child software.
Clarified the built-in required vs. optional args by allowing pipeline authors to specify which of the pypiper args are required. The command-line help UI now displays these correctly as 'required arguments' instead of incorrectly as 'optional arguments'.
Corrected the sort order of added arguments, so they are listed in the help menu more naturally.
Fixed a bug that caused an erroneous error message indicating missing pypiper args.
Clarified the license is BSD2
Fixed a bug that neglected to list pyyaml as a dependency
v0.7.2 (2018-06-05):
Implemented the 'report object' function.
Cleanup files are now relative, so a moved folder could still be cleaned.
Fixed a bug that prevented install if pypandoc was not installed
Fixed a bug that caused an error in containers where /proc wasn't accessible
v0.7.1 (2018-02-27):
v0.7.0 (2017-12-12):
Standardize NGSTk function naming.
Introduce Stage as a model for a logically related set of pipeline processing steps.
Introduce Pipeline framework for automated processing phase execution and checkpointing.
Add ability to start and/or stop a pipeline at arbitrary checkpoints.
Introduce new state for a paused/halted pipeline.
Improve spawned process shutdown to avoid zombie processes.
v0.6 (2017-08-24):
Adds 'dynamic recovery' capability. For jobs that are terminated by an interrupt, such as a SIGINT or SIGTERM (as opposed to a failed command), pypiper will now set a dynamic recovery flags. These jobs, when restarted, will automatically pick up where they left off, without requiring any user intervention. Previously, the user would have to specify recover mode (-R). Now, recover mode forces a recover regardless of failure type, but interrupted pipelines will auto-recover.
Pypiper now appropriately adds cleanup files intermediate files for failed runs. It adds them to the cleanup script.
Improves error messages so only a single exception is raised with a more direct relevance to the user/
Pypiper will automatically remove existing flags when the run starts, eliminating the earlier issue of confusion due to multiple flags present on runs that were restarted.
Fixes a bug that caused a pipeline to continue if a SIGTERM is given during a process that was marked nofail.
Pypiper now can handle multiple SIGTERMs without one canceling the shutdown procedure begun by the other.
Major improvements to documentation and tutorials.
Adds report_figure function.
v0.5 (2017-07-21):
Adds preliminary support for handling docker containers
Updates docs, adds Hello World example
Adds 'waiting' flag
Eliminates extra spaces in reported results
Pypiper module is version aware
Updates Success time format to eliminate space
Improves efficiency in some ngstk merging functions
v0.4 (2017-01-23):
First major public release!
Revamps pypiper args
Adds parallel compression/decompression with pigz
Various small bug fixes and speed improvements