1 of 8

CHIP Variant Calling and Processing

Clonal Hematopoesis Varaint Calling and Process

Workflow that generates VCF files, which are then annotated and filtered for CMO-CH analysis.

Introduction

CHIP-VAR pipeline generates VCF and MAF files from the input BAM file which are then processed by the single sample method, annotated by multiple databases and tools, and finally filtered to generate high-confidence Clonal Hematopoetic Putative-Driver (CHPD) calls. Detailed descriptions of the process, tools, and output can be found in this gitbook.

Credits

CMO cfDNA Informatics Team

Workflow Description

CHIP-VAR workflow consists of multiple sub-workflows, written in Common Workflow Language (CWL). Figure 1 shows the complete pipeline grouped according to functionality.

The 3 major steps in the order of working are

Variant Calling,
Variant Annotation,

Requirements

The following requirements need to be fulfilled before running the pipeline.

A system with either docker or singularity configured.
Python 3.6 and above (for running and running )
- Python Packages :
  - toil[cwl]

Installation and Usage

If you have paired-end umi-tagged fastqs, you can run the ACCESS fastq to bam workflow with the following steps

Step 1: Create a virtual environment.

Option (A) - if using cwltool

If you are using cwltool only, please proceed using python 3.9 as done below:

Here we can use either or . Here we will use conda.

Option (B) - recommended for Juno HPC cluster

If you are using toil, python 3 is required. Please install using Python 3.9 as done below:

Here we can use either or . Here we will use conda.

Once you execute the above command you will see your bash prompt something on this lines:

Step 2: Clone the repository

Note: Change 3.0.4 to the latest stable release of the pipeline

Step 3: Install requirements using pip

We have already specified the version of cwltool and other packages in the requirements.txt file. Please use this to install.

Step 4: Check if you have singularity and nodejs for HPC

For HPC normally singularity is used for containers. Thus please make sure that is installed. For JUNO, you can do the following:

We also need to make sure nodejs is installed, this can be installed using conda:

Step 5: Generate an inputs file

Next, you must generate a proper input file in either or format.

For details on how to create this file, please follow this example (there is a minimal example of what needs to be filled in at the end of the page):

It's also possible to create and fill in a "template" inputs file using this command:

This may or may not work. We are not exactly sure why. But you can always use Rabix to generate the template input

Note: To see help for the inputs for cwl workflow you can use: toil-cwl-runner chip-var.cwl --help

Once we have successfully installed the requirements we can now run the workflow using cwltool/toil .

Step 6: Run the workflow

Here we show how to use to run the workflow on a single machine, such as a laptop

Run the workflow with a given set of input using on single machine

Here we show how to run the workflow using using single machine interface

Once we have successfully installed the requirements we can now run the workflow using

Usage

Tools Description

Versions of tools in order of process

Tool

Version

1.8.2

0.1.5

1.15.1

1.6

Files and Resources used

There are multiple files from different resources used in this workflow.

Inputs Description

Input files and parameters required to run workflow

Common Parameters

Output Description

Files present after workflow is finished

Output File

Description

sample-name_vardict_STDFilter.txt

TXT file containing basic information on calls

sample-name_single_filter_vcf

VCF file with filtered SNPs

sample-name_single_filer_complex.vcf

VCF file with filtered complex variants

sample-name_vardict_concatenated.vcf

Usage of individual commands

Vardict v1.8.2

/usr/bin/vardict/bin/VarDict -f "0" \
    -c "1" \
    -g "5" \
    -E "3" \
    -S "2" \
    -G /path/to/reference_fasta.ext \
    -N sample_name \
    -b /path/to/bam/file.bam \
    -bedfile /path/to/bedfile.ext \
    > vardict_app_output.vcf

Rscript /usr/bin/vardict/bin/teststrandbias.R > output_teststrandbias.var \
    < /path/to/input_vardict.var

pv vardict v0.1.5

BCFTOOLS v 1.15.1

BGZIP

TABIX

BCFTOOLS SORT

BCFTOOLS NORM

BCFTOOLS CONCAT

SNPSIFT annotation v5.0:

vcf2maf v1.6.21

oncoKB annotator v3.2.2

PV modules v0.2.3

MAF annotate by BED

MAF annotate by TSV

MAF tag

MAF filter

reference_fasta: 
 class: File
 path: >-
    /juno/work/access/production/resources/reference/current/Homo_sapiens_assembly19.fasta
bedfile:
  class: File
  path: >-
    /work/bergerm1/bergerlab/charalk/projects/nucleo_qc/qc_generation_testing/CH_target_3bp.bed
input_bam_case: 
  class: File
  path: >-
    /path/to/bam/file.bam
input_cosmicCountDB_vcf: 
  class: File
  path: >-
    /work/cch/production/resources/cosmic/versions/v96/CosmicCodingMuts.vcf.gz
input_cosmicprevalenceDB_vcf: 
  class: File
  path: >-
    /work/cch/production/resources/cosmic/versions/v96/CosmicCodingMuts_GRCh37_processed.vcf.gz
input_complexity_bed: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/rmsk_mod.bed
input_mappability_bed: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/wgEncodeDacMapabilityConsensusExcludable_4cols.bed
oncoKbApiToken: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/apiToken.txt
input_47kchpd_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/chpd47k_prevalence.tsv
input_hotspot_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/hotspots_47kchpd.tsv
input_panmeloid_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/pan_myeloid_ks.tsv 
opOncoKbMafName: sampleName_oncokb.maf
output_complexity_filename: sampleName_complexity.maf
output_mappability_filename: sampleName_mappability.maf
output_vcf2mafName: sampleName_vcf2maf.maf
concat_output_name: sampleName_concat.vcf.gz
retain_info:  CNT,TUMOR_TYPE
sample_name: sampleName
vardict_allele_frequency_threshold: 0
vardict_output_vcf_name: sampleName_vardict.vcf
snpsift_countOpName: sampleName_snpsift_cosmic.vcf
snpsift_prevalOpName: sampleName_snpsift_preval.vcf
column_name_complexity: complexity
column_name_mappability: mapability

Installation and Usage

If you have paired-end umi-tagged fastqs, you can run the ACCESS fastq to bam workflow with the following steps

Step 1: Create a virtual environment.

Option (A) - if using cwltool

If you are using cwltool only, please proceed using python 3.9 as done below:

Here we can use either or . Here we will use conda.

Option (B) - recommended for Juno HPC cluster

If you are using toil, python 3 is required. Please install using Python 3.9 as done below:

Here we can use either or . Here we will use conda.

Once you execute the above command you will see your bash prompt something on this lines:

Step 2: Clone the repository

Note: Change 3.0.4 to the latest stable release of the pipeline

Step 3: Install requirements using pip

We have already specified the version of cwltool and other packages in the requirements.txt file. Please use this to install.

Step 4: Check if you have singularity and nodejs for HPC

For HPC normally singularity is used for containers. Thus please make sure that is installed. For JUNO, you can do the following:

We also need to make sure nodejs is installed, this can be installed using conda:

Step 5: Generate an inputs file

Next, you must generate a proper input file in either or format.

For details on how to create this file, please follow this example (there is a minimal example of what needs to be filled in at the end of the page):

It's also possible to create and fill in a "template" inputs file using this command:

This may or may not work. We are not exactly sure why. But you can always use Rabix to generate the template input

Note: To see help for the inputs for cwl workflow you can use: toil-cwl-runner chip-var.cwl --help

Once we have successfully installed the requirements we can now run the workflow using cwltool/toil .

Step 6: Run the workflow

Here we show how to use to run the workflow on a single machine, such as a laptop

Run the workflow with a given set of input using on single machine

Here we show how to run the workflow using using single machine interface

Once we have successfully installed the requirements we can now run the workflow using

Usage

Usage of individual commands

Vardict v1.8.2

/usr/bin/vardict/bin/VarDict -f "0" \
    -c "1" \
    -g "5" \
    -E "3" \
    -S "2" \
    -G /path/to/reference_fasta.ext \
    -N sample_name \
    -b /path/to/bam/file.bam \
    -bedfile /path/to/bedfile.ext \
    > vardict_app_output.vcf

Rscript /usr/bin/vardict/bin/teststrandbias.R > output_teststrandbias.var \
    < /path/to/input_vardict.var

perl /usr/bin/vardict/bin/var2vcf_valid.pl \\
 -N sample_name
 -f 0
 > output_vcf_name
 < /path/to/input_vardict.var

pv vardict v0.1.5

pv vardict single filter -i /path/to/input_vardict.vcf \
    --tsampleName sample-name \
    -ad 1 \
    -fg false \
    -mq 0 \
    -tnr 1 \
    -dp 20 \
    -vf 5e-05

BCFTOOLS v 1.15.1

BGZIP

bgzip -c /path/to/input_single_filter.vcf > sample_name.vcf.gz

TABIX

tabix -p vcf sample_name.vcf.gz

BCFTOOLS SORT

#BCFTOOLS SORT
bcftools sort -O z -o sample_name_sorted.vcf.gz sample_name.vcf.gz

BCFTOOLS NORM

bcftools norm  --check-ref s \
  -m + \
  -O z \
  -o sample_name_norm.vcf.gz \
  -f /path/to/ref_fasta.fa \
  sample_name_sorted.vcf.gz

BCFTOOLS CONCAT

SNPSIFT annotation v5.0:

vcf2maf v1.6.21

oncoKB annotator v3.2.2

PV modules v0.2.3

MAF annotate by BED

MAF annotate by TSV

MAF tag

MAF filter

reference_fasta: 
 class: File
 path: >-
    /juno/work/access/production/resources/reference/current/Homo_sapiens_assembly19.fasta
bedfile:
  class: File
  path: >-
    /work/bergerm1/bergerlab/charalk/projects/nucleo_qc/qc_generation_testing/CH_target_3bp.bed
input_bam_case: 
  class: File
  path: >-
    /path/to/bam/file.bam
input_cosmicCountDB_vcf: 
  class: File
  path: >-
    /work/cch/production/resources/cosmic/versions/v96/CosmicCodingMuts.vcf.gz
input_cosmicprevalenceDB_vcf: 
  class: File
  path: >-
    /work/cch/production/resources/cosmic/versions/v96/CosmicCodingMuts_GRCh37_processed.vcf.gz
input_complexity_bed: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/rmsk_mod.bed
input_mappability_bed: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/wgEncodeDacMapabilityConsensusExcludable_4cols.bed
oncoKbApiToken: 
  class: File
  path:  >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/apiToken.txt
input_47kchpd_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/chpd47k_prevalence.tsv
input_hotspot_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/hotspots_47kchpd.tsv
input_panmeloid_tsv_file: 
  class: File
  path: >-
    /work/bergerm1/bergerlab/sivaprk/chipvar_resources/pan_myeloid_ks.tsv 
opOncoKbMafName: sampleName_oncokb.maf
output_complexity_filename: sampleName_complexity.maf
output_mappability_filename: sampleName_mappability.maf
output_vcf2mafName: sampleName_vcf2maf.maf
concat_output_name: sampleName_concat.vcf.gz
retain_info:  CNT,TUMOR_TYPE
sample_name: sampleName
vardict_allele_frequency_threshold: 0
vardict_output_vcf_name: sampleName_vardict.vcf
snpsift_countOpName: sampleName_snpsift_cosmic.vcf
snpsift_prevalOpName: sampleName_snpsift_preval.vcf
column_name_complexity: complexity
column_name_mappability: mapability

usage: chip-var.cwl [-h] --reference_fasta REFERENCE_FASTA --input_bam_case INPUT_BAM_CASE
                    [--bedfile BEDFILE] --sample_name SAMPLE_NAME
                    [--vardict_allele_frequency_threshold VARDICT_ALLELE_FREQUENCY_THRESHOLD]
                    [--retain_info RETAIN_INFO] --concat_output_name CONCAT_OUTPUT_NAME
                    [--vardict_output_vcf_name VARDICT_OUTPUT_VCF_NAME]
                    --input_cosmicprevalenceDB_vcf INPUT_COSMICPREVALENCEDB_VCF
                    --input_cosmicCountDB_vcf INPUT_COSMICCOUNTDB_VCF
                    [--snpsift_prevalOpName SNPSIFT_PREVALOPNAME]
                    [--snpsift_countOpName SNPSIFT_COUNTOPNAME] --input_complexity_bed
                    INPUT_COMPLEXITY_BED
                    [--output_complexity_filename OUTPUT_COMPLEXITY_FILENAME]
                    [--column_name_complexity COLUMN_NAME_COMPLEXITY] --oncoKbApiToken
                    ONCOKBAPITOKEN --opOncoKbMafName OPONCOKBMAFNAME
                    [--output_vcf2mafName OUTPUT_VCF2MAFNAME] --input_mappability_bed
                    INPUT_MAPPABILITY_BED
                    [--output_mappability_filename OUTPUT_MAPPABILITY_FILENAME]
                    [--column_name_mappability COLUMN_NAME_MAPPABILITY]
                    --input_47kchpd_tsv_file INPUT_47KCHPD_TSV_FILE --input_hotspot_tsv_file
                    INPUT_HOTSPOT_TSV_FILE --input_panmeloid_tsv_file INPUT_PANMELOID_TSV_FILE
                    [job_order]

chip-var

positional arguments:
  job_order             Job input json file

options:
  -h, --help            show this help message and exit
  --reference_fasta REFERENCE_FASTA
  --input_bam_case INPUT_BAM_CASE
  --bedfile BEDFILE
  --sample_name SAMPLE_NAME
  --vardict_allele_frequency_threshold VARDICT_ALLELE_FREQUENCY_THRESHOLD
  --retain_info RETAIN_INFO
  --concat_output_name CONCAT_OUTPUT_NAME
  --vardict_output_vcf_name VARDICT_OUTPUT_VCF_NAME
  --input_cosmicprevalenceDB_vcf INPUT_COSMICPREVALENCEDB_VCF
  --input_cosmicCountDB_vcf INPUT_COSMICCOUNTDB_VCF
  --snpsift_prevalOpName SNPSIFT_PREVALOPNAME
  --snpsift_countOpName SNPSIFT_COUNTOPNAME
  --input_complexity_bed INPUT_COMPLEXITY_BED
  --output_complexity_filename OUTPUT_COMPLEXITY_FILENAME
  --column_name_complexity COLUMN_NAME_COMPLEXITY
  --oncoKbApiToken ONCOKBAPITOKEN
  --opOncoKbMafName OPONCOKBMAFNAME
  --output_vcf2mafName OUTPUT_VCF2MAFNAME
  --input_mappability_bed INPUT_MAPPABILITY_BED
  --output_mappability_filename OUTPUT_MAPPABILITY_FILENAME
  --column_name_mappability COLUMN_NAME_MAPPABILITY
  --input_47kchpd_tsv_file INPUT_47KCHPD_TSV_FILE
  --input_hotspot_tsv_file INPUT_HOTSPOT_TSV_FILE
  --input_panmeloid_tsv_file INPUT_PANMELOID_TSV_FILE

CHIP Variant Calling and Processing

Clonal Hematopoesis Varaint Calling and Process

hashtagIntroduction

hashtagCredits

Workflow Description

Requirements

hashtagThe following requirements need to be fulfilled before running the pipeline.

Installation and Usage

hashtagStep 1: Create a virtual environment.

hashtagOption (A) - if using cwltool

hashtagOption (B) - recommended for Juno HPC cluster

hashtagStep 2: Clone the repository

hashtagStep 3: Install requirements using pip

hashtagStep 4: Check if you have singularity and nodejs for HPC

hashtagStep 5: Generate an inputs file

hashtagStep 6: Run the workflow

hashtagRun the workflow with a given set of input using on single machine

hashtagUsage

Tools Description

Inputs Description

hashtagCommon Parameters

Output Description

Usage of individual commands

Requirements

hashtagThe following requirements need to be fulfilled before running the pipeline.

Clonal Hematopoesis Varaint Calling and Process

hashtagIntroduction

hashtagCredits

Tools Description

Inputs Description

hashtagCommon Parameters

Workflow Description

hashtagExample Input YML file to run the CWL

hashtag

Installation and Usage

hashtagStep 1: Create a virtual environment.

hashtagOption (A) - if using cwltool

hashtagOption (B) - recommended for Juno HPC cluster

hashtagStep 2: Clone the repository

hashtagStep 3: Install requirements using pip

hashtagStep 4: Check if you have singularity and nodejs for HPC

hashtagStep 5: Generate an inputs file

hashtagStep 6: Run the workflow

hashtagRun the workflow with a given set of input using on single machine

hashtagUsage

Output Description

Usage of individual commands

hashtagRun the workflow with a given set of input using toilarrow-up-right on single machine

Introduction

Credits

The following requirements need to be fulfilled before running the pipeline.

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Option (B) - recommended for Juno HPC cluster

Step 2: Clone the repository

Step 3: Install requirements using pip

Step 4: Check if you have singularity and nodejs for HPC

Step 5: Generate an inputs file

Step 6: Run the workflow

Run the workflow with a given set of input using on single machine

Usage

Common Parameters

The following requirements need to be fulfilled before running the pipeline.

Introduction

Credits

Common Parameters

Example Input YML file to run the CWL

Step 1: Create a virtual environment.

Option (A) - if using cwltool

Option (B) - recommended for Juno HPC cluster

Step 2: Clone the repository

Step 3: Install requirements using pip

Step 4: Check if you have singularity and nodejs for HPC

Step 5: Generate an inputs file

Step 6: Run the workflow

Run the workflow with a given set of input using on single machine

Usage

Run the workflow with a given set of input using toil on single machine