FancyPipe is an open source python package for running cpu-intensive data processing pipelines. It covers those aspects that would otherwise distract you from your main job: the design of the processing steps.
FancyPipe takes care of:
- parallel execution of code
- html-based, hierarchical log file
- parameter management
- command line interface
- temporary files and garbage collection
- error handling
More details in the documentation.