Link Search Menu Expand Document

Pipeline Actions

Stage 1 Pipelines each have a set of actions - the staged units of work they are intended to accomplish, encoded in the pipeline’s configuration - and subcommands - pipeline-level management actions handled by the MDI pipelines framework.

For example, here is the inline help for the svCapture pipeline.

$ mdi svCapture

>>> Michigan Data Interface (MDI): Stage 1 Pipelines <<<

svCapture: Characterize structural variant junctions in short-read, paired-end capture library(s)

usage:
  mdi svCapture <data.yml> [options]  # run all pipeline actions in data.yml
  mdi svCapture <action> <data.yml> [options] # run one action from data.yml
  mdi svCapture <action> <options>    # run one action, all options from command line
  mdi svCapture <action> --help       # pipeline action help
  mdi svCapture --help                # summarize pipeline actions

pipeline specific actions:
  align          clean paired-end reads and align to reference genome; output name-sorted bam/cram
  collate        collate read groups, make consensuses, and re-align to genome
  extract        scan name-sorted reads for molecules with alignment discontinuities
  find           scan anomalous molecules from one or more samples to make SV calls
  genotype       genotype the bulk capture regions and score SNVs in SV junction sequences

general workflow commands:
  template       return a template for creating an input-specific `data.yml` file
  conda          create, update, or list the conda environment(s) required by a pipeline
  build          build a pipeline Singularity image and push to a registry
  shell          open a command shell in a pipeline action's runtime environment
  status         print the pipeline status file for an output directory
  rollback       revert the pipeline to an earlier step for an output directory

Pipeline actions

When you make a call to a pipeline, you might queue multiple actions listed in data.yml, or you might need to run just one action at a time, denoted as <action> above.

By default, when a pipeline has just one action it is called ‘do’, leading to calls like:

mdi myPipeline do data.yml

Job configuration options and templates

We strongly recommend using data.yml files to enter the sometimes numerous options for the work you will execute. The template subcommand makes it easy to create a pre-assembled job configuration file for a pipeline for you to modify.

Job runtime environments

The MDI places a premium on consistent, controlled execution environments by using conda. It goes like this.

Developers list the software their pipeline needs in their configuration files.

End users make calls to the conda subcommand to build a pipeline’s environment(s) prior to running any work. No other software installation is needed!

Alternatively, developers make calls to the build subcommand to create and upload pre-assembled Singularity containers carrying the required environment(s) and mark their pipelines as supporting containers for users to download. Users must have Singularity available on their server capable of running (but not building) containers.

Because most shared HPC servers do not support Singularity builds, the MDI provides resources that make it easy for developers to build containers on Amazon Web Services: