1 of 4

Installation and Running

Workflows that generate, aggregate, and visualize quality control files for MSK-ACCESS.

Given the output files from Nucleo, there are workflows to generate the quality control files, aggregate them files across many samples, and visualize them using MultQC. You can choose to run these workflows whether you have just one or hundreds of samples. Depending on your use case, there are two main options:

(1) Run qc_generator.cwl followed by aggregate_visualize.cwl. This approach first generates the QC files for one or more samples, and you use the second CWL script to aggregate the QC files and visualize them with MultiQC. This option can be useful for when you want to generate the QC files for some samples just once and then reuse those samples in multiple MultiQC reports.

(2) Run just access_qc.cwl. This option just combines the two steps from the first option into one workflow. This workflow can

Warning: Including more than 50 samples in the MultiQC report will cause some figures to lose interactivity. Including more than a few hundreds samples may cause MultiQC to fail.

Requirements

Before Installation of the pipeline, make sure your system supports these requirements

Following are the requirements for running the workflow:

A system with either or configured.
Python 3.6 (for running and running )
- Python Packages (will be installed as part of pipeline installation):

Installation and Usage

You must have run the Nucleo workflow first before running any of the MSK-ACCESS QC workflows. Depending on your use case, there are two main sets of workflows you can choose to run: (1) `qc_generator

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Inputs Description

Input files and parameters required to run workflow

Common workflow language execution engines accept two types of input that are JSON or YAML, please make sure to use one of these while generating the input file. For more information refer to: http://www.commonwl.org/user_guide/yaml/

Parameter Used by Tools

Common Parameters Across Tools

Template Inputs File

toil[cwl]==5.1.0

pytz==2021.1

typing==3.7.4.3

ruamel.yaml==0.16.5

pip==20.2.3

bumpversion==0.6.0

wheel==0.35.1

watchdog==0.10.3

flake8==3.8.4

tox==3.20.0

coverage==5.3

twine==3.2.0

pytest==6.1.1

pytest-runner==5.2

coloredlogs==10.0

pytest-travis-fold==1.3.0

Installation and Running

Workflows that generate, aggregate, and visualize quality control files for MSK-ACCESS.

(2) Run just access_qc.cwl. This option just combines the two steps from the first option into one workflow. This workflow can

Warning: Including more than 50 samples in the MultiQC report will cause some figures to lose interactivity. Including more than a few hundreds samples may cause MultiQC to fail.

inputs.yaml

biometrics_bed_file:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-B.sorted.bed
biometrics_json: true
biometrics_plot: true
biometrics_vcf_file:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-TilingaAndFpSNPs.vcf
collapsed_bam:
- class: File
  path: /path/to/bam
collapsed_biometrics_coverage_threshold: null
collapsed_biometrics_major_threshold: null
collapsed_biometrics_min_base_quality: null
collapsed_biometrics_min_coverage: null
collapsed_biometrics_min_homozygous_thresh: null
collapsed_biometrics_min_mapping_quality: null
collapsed_biometrics_minor_threshold: null
duplex_bam:
- class: File
  path: /path/to/bam
duplex_biometrics_major_threshold: null
duplex_biometrics_min_base_quality: null
duplex_biometrics_min_coverage: null
duplex_biometrics_min_homozygous_thresh: null
duplex_biometrics_min_mapping_quality: null
duplex_biometrics_minor_threshold: null
group_reads_by_umi_bam:
- class: File
  path: /path/to/bam
hsmetrics_coverage_cap: 30000
hsmetrics_minimum_base_quality: 1
hsmetrics_minimum_mapping_quality: 1
noise_sites_bed:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-A_no_msi_sorted_deduped.bed
pool_a_bait_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-A_baits.sorted.interval_list
pool_a_target_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0_panelA_targets.interval_list
pool_b_bait_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-B_baits.sorted.interval_list
pool_b_target_intervals:
  class: File
  path: /path
reference:
  class: File
  path: /path
sample_group:
- patient_id
sample_name:
- sample_id
sample_sex:
- M
sample_type:
- tumor
sequence_qc_min_basq: 1
sequence_qc_min_mapq: 1
sequence_qc_threshold: null
sequence_qc_truncate: null
simplex_bam:
- class: File
  path: /path
uncollapsed_bam:
- class: File
  path: /path
uncollapsed_bam_base_recal:
- class: File
  path: /path

inputs.yaml

biometrics_bed_file:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-B.sorted.bed
biometrics_json: true
biometrics_plot: true
biometrics_vcf_file:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-TilingaAndFpSNPs.vcf
collapsed_bam:
- class: File
  path: /path/to/bam
collapsed_biometrics_coverage_threshold: null
collapsed_biometrics_major_threshold: null
collapsed_biometrics_min_base_quality: null
collapsed_biometrics_min_coverage: null
collapsed_biometrics_min_homozygous_thresh: null
collapsed_biometrics_min_mapping_quality: null
collapsed_biometrics_minor_threshold: null
duplex_bam:
- class: File
  path: /path/to/bam
duplex_biometrics_major_threshold: null
duplex_biometrics_min_base_quality: null
duplex_biometrics_min_coverage: null
duplex_biometrics_min_homozygous_thresh: null
duplex_biometrics_min_mapping_quality: null
duplex_biometrics_minor_threshold: null
group_reads_by_umi_bam:
- class: File
  path: /path/to/bam
hsmetrics_coverage_cap: 30000
hsmetrics_minimum_base_quality: 1
hsmetrics_minimum_mapping_quality: 1
noise_sites_bed:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-A_no_msi_sorted_deduped.bed
pool_a_bait_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-A_baits.sorted.interval_list
pool_a_target_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0_panelA_targets.interval_list
pool_b_bait_intervals:
  class: File
  path: /path/to/MSK-ACCESS-v1_0-probe-B_baits.sorted.interval_list
pool_b_target_intervals:
  class: File
  path: /path
reference:
  class: File
  path: /path
sample_group:
- patient_id
sample_name:
- sample_id
sample_sex:
- M
sample_type:
- tumor
sequence_qc_min_basq: 1
sequence_qc_min_mapq: 1
sequence_qc_threshold: null
sequence_qc_truncate: null
simplex_bam:
- class: File
  path: /path
uncollapsed_bam:
- class: File
  path: /path
uncollapsed_bam_base_recal:
- class: File
  path: /path

Here we show how to use cwltool to run the workflow on a single machine, such as a laptop

Run the workflow with a given set of input using cwltool on single machine

To generate the QC files for one sample:

cwltool-execution

cwltool nucleo.cwl inputs.yaml

To aggregate the QC files across one or more samples and visualize with MultiQC:

Here we show how to run the workflow using using single machine interface

Once we have successfully installed the requirements we can now run the workflow using cwltool if you have proper input file generated either in or format. Please look at for more details.

Run the workflow with a given set of input using on single machine

Here we show how to run the workflow using on MSKCC internal compute cluster called JUNO which has as a scheduler.

Note the use of --singularityto convert Docker containers into singularity containers, the TMPDIR environment variable to avoid writing temporary files to shared disk space, the _JAVA_OPTIONS environment variable to specify java temporary directory to /scratch, using SINGULARITY_BINDPATH environment variable to bind the /scratch when running singularity containers and TOIl_LSF_ARGS to specify any additional arguments to bsub

toil-lsf-execution

TMPDIR=$PWD
TOIL_LSF_ARGS='-W 3600 -P test_nucleo -app anyOS -R select[type==CentOS7]'
_JAVA_OPTIONS='-Djava.io.tmpdir=/scratch/'
SINGULARITY_BINDPATH='/scratch:/scratch:rw'
toil-cwl-runner \
       --singularity \
       --logFile ./example.log  \
       --jobStore ./example_jobStore \
       --batchSystem lsf \
       --workDir ./example_working_directory/ \
       --outdir $PWD \
       --writeLogs ./example_log_folder/ \
       --logLevel DEBUG \
       --stats \
       --retryCount 2 \
       --disableCaching \
       --disableChaining \
       --preserve-environment TOIL_LSF_ARGS TMPDIR \
       --maxLogFileSize 20000000000 \
       --cleanWorkDir onSuccess \
       nucleo.cwl \
       inputs.yaml \
       > toil.stdout \
       2> toil.stderr &

toil-lsf-execution

TMPDIR=$PWD
TOIL_LSF_ARGS='-W 3600 -P test_nucleo -app anyOS -R select[type==CentOS7]'
_JAVA_OPTIONS='-Djava.io.tmpdir=/scratch/'
SINGULARITY_BINDPATH='/scratch:/scratch:rw'
toil-cwl-runner \
       --singularity \
       --logFile ./example.log  \
       --jobStore ./example_jobStore \
       --batchSystem lsf \
       --workDir ./example_working_directory/ \
       --outdir $PWD \
       --writeLogs ./example_log_folder/ \
       --logLevel DEBUG \
       --stats \
       --retryCount 2 \
       --disableCaching \
       --disableChaining \
       --preserve-environment TOIL_LSF_ARGS TMPDIR \
       --maxLogFileSize 20000000000 \
       --cleanWorkDir onSuccess \
       nucleo.cwl \
       inputs.yaml \
       > toil.stdout \
       2> toil.stderr &

Installation and Running

Requirements

Requirements

Before Installation of the pipeline, make sure your system supports these requirements

Installation and Usage

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Inputs Description

Parameter Used by Tools

Common Parameters Across Tools

Template Inputs File

Requirements

Requirements

Before Installation of the pipeline, make sure your system supports these requirements

Installation and Running

Inputs Description

Parameter Used by Tools

Common Parameters Across Tools

Template Inputs File

Installation and Usage

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Option (B) - recommended for Juno HPC cluster

Step 2: Clone the repository

Step 3: Install requirements using pip

Step 4: Generate an inputs file

Step 5: Run the workflow

Run the workflow with a given set of input using cwltool on single machine

Run the workflow with a given set of input using on single machine

Installation and Running

Requirements

hashtagRequirements

hashtagBefore Installationarrow-up-right of the pipeline, make sure your system supports these requirements

Installation and Usage

hashtagStep 1: Create a virtual environment.

hashtagOption (A) - if using cwltool

Inputs Description

hashtagParameter Used by Tools

hashtagCommon Parameters Across Tools

hashtagTemplate Inputs File

Requirements

hashtagRequirements

hashtagBefore Installationarrow-up-right of the pipeline, make sure your system supports these requirements

Installation and Running

Inputs Description

hashtagParameter Used by Tools

hashtagCommon Parameters Across Tools

hashtagTemplate Inputs File

Installation and Usage

hashtagStep 1: Create a virtual environment.

hashtagOption (A) - if using cwltool

hashtagOption (B) - recommended for Juno HPC cluster

hashtagStep 2: Clone the repository

hashtagStep 3: Install requirements using pip

hashtagStep 4: Generate an inputs file

hashtagStep 5: Run the workflow

hashtagRun the workflow with a given set of input using cwltoolarrow-up-right on single machine

hashtagRun the workflow with a given set of input using on single machine

Requirements

Before Installation of the pipeline, make sure your system supports these requirements

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Parameter Used by Tools

Common Parameters Across Tools

Template Inputs File

Requirements

Before Installation of the pipeline, make sure your system supports these requirements

Parameter Used by Tools

Common Parameters Across Tools

Template Inputs File

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Option (B) - recommended for Juno HPC cluster

Step 2: Clone the repository

Step 3: Install requirements using pip

Step 4: Generate an inputs file

Step 5: Run the workflow

Run the workflow with a given set of input using cwltool on single machine

Run the workflow with a given set of input using on single machine