Differences

This shows you the differences between two versions of the page.

--- howto:bioinformatics [2025/05/01 23:20]
nmfuphi
+++ howto:bioinformatics [2025/05/21 10:50] (current)
nmfuphi [Singularity]
@@ Line 195: / Line 195: @@
 ===== Software Environments =====
+==== Conda ====
+Many scientific software tools rely on specific versions of libraries, compilers, and dependencies that often conflict with each other or with system-wide installations. **Conda** is a powerful, language-agnostic environment and package manager that helps solve this problem by allowing users to manage **Python**, **R**, **C/C++**, **FORTRAN**, and other language ecosystems in **isolated environments**.
+=== Shared Conda Environments ===
+For most use cases, especially in bioinformatics, CHPC provides pre-built, **shared Conda environments** installed under:
+  '/apps/chpc/bio/anaconda3-2020.02/envs'
+These environments are curated by CHPC staff to include commonly used tools in genomics, transcriptomics, and other domains.
+=== Step-by-step usage ===
+=== Step 1: Load required modules ===
+To access Conda functionality, first load the required modules:
+<code bash>
+module load chpc/BIOMODULES
+module load conda_init
+</code>
+The second module updates your .bashrc file by adding necessary shell variables.
+To apply these changes, you can either log out and log back in, or run: <code bash>source ~/.bashrc. </code>
+After this setup, you won’t need to load additional modules for your jobs—only the eval and conda activate steps are required.
+=== Step 2: Initialize Conda in your shell ===
+Activate Conda shell integration:
+<code bash>
+eval "$(conda shell.bash hook)"
+</code>
+This command sets up your shell environment to recognize Conda commands like `conda activate`.
+=== Step 3: List available environments ===
+<code bash>
+conda info --envs
+</code>
+This will display all available shared Conda environments and their paths.
+=== Step 4: Activate a shared environment ===
+<code bash>
+conda activate nameOfTheEnv
+</code>
+Replace ''nameOfTheEnv'' with the name of an environment from the previous step.
+> **Tip:** If you're unsure which environment to use, contact CHPC support or explore the environment's contents with `conda list`.
+> **Note:** You do **not** need to install anything when using shared environments.
+=== Creating Private Conda Environments ===
+If you need software that is not included in the shared environments, you may create your own **private Conda environment**. This gives you full control over the software stack and package versions.
+> **Important:** Do **not** install environments in your home directory (''/home/<username>'') -use your Lustre project storage instead.
+=== Step-by-step setup ===
+ssh to username@scp.chpc.ac.za, the password is the same as the one you use on lengau
+=== Step 1: Load Conda ===
+<code bash>
+module load chpc/BIOMODULES
+module load conda_init
+eval "$(conda shell.bash hook)"
+</code>
+=== Step 2: Create a new environment ===
+<code bash>
+conda create --prefix /mnt/lustre/<username>/myenv python=3.10
+</code>
+This will create a Conda environment at the specified path with Python 3.10 installed. You can replace the Python version or leave it out if not needed.
+=== Step 3: Activate your environment ===
+<code bash>
+conda activate /mnt/lustre/<username>/myenv
+</code>
+After activation, you can install any packages you need.
+=== Step 4: (Optional) Install Mamba for faster package management ===
+<code bash>
+conda install mamba -n base -c conda-forge
+</code>
+> **Tip:** Mamba is a drop-in replacement for Conda that uses a faster dependency solver written in C++. Once installed, you can use `mamba` instead of `conda` for installing packages:
+<code bash>
+mamba install numpy pandas
+</code>
+This significantly speeds up installations and environment solves, especially when working with large scientific packages.
+=== Step 5: Install packages ===
+<code bash>
+conda install numpy pandas matplotlib
+</code>
+You can install packages one by one, or include them during environment creation:
+<code bash>
+conda create --prefix /mnt/lustre/<username>/myenv python=3.10 numpy pandas
+</code>
+=== Step 6: Remove unused environments ===
+Old or unused environments can be removed to free up space:
+<code bash>
+conda remove --prefix /mnt/lustre/<username>/myenv --all
+</code>
+=== Best Practices ===
+  * ✅ Use **shared environments** whenever possible for consistency and faster setup.
+  * 📁 Create private environments **only in Lustre** directories, such as ''/mnt/lustre/<username>''.
+  * ⚠️ Do **not** use Conda in your ''$HOME'' directory, it may lead to quota issues or slow performance.
+  * 📌 Use the ''--prefix'' flag to create environments with absolute paths, especially on clusters where ''--name'' may default to ''$HOME''.
+  * 🧼 Periodically clean up unused environments with `conda remove --all`.
+  * 🔁 Reuse environment definitions by exporting and sharing them with others or for reproducibility.
+=== Troubleshooting ===
+  * ❓ **Conda not recognized?** Make sure you loaded `conda_init` and ran `eval "$(conda shell.bash hook)"`.
+  * 🚫 **Permission denied?** You might be trying to write to a restricted directory like ''/apps'' or ''$HOME''.
+  * 🔄 **Environment behaving unexpectedly?** Try deactivating (`conda deactivate`) and reactivating, or recreate the environment.
+  * 🧪 **Conflicts during install?** Use `conda clean --all` to clear caches and retry with a minimal environment.
 ==== Singularity ====
@@ Line 340: / Line 462: @@
 #PBS -N singularity_job
 #PBS -q normal
-#PBS -l select=1:ncpus=8:mem=32gb
+#PBS -l select=1:ncpus=24
 #PBS -l walltime=12:00:00
 #PBS -o singularity_output.log
@@ Line 357: / Line 479: @@
 </code>
+==== Nextflow ====
-**Need Help?**
+Nextflow is a free and open-source workflow management system that enables the development and execution of data analysis pipelines. It simplifies complex computational workflows and ensures reproducibility, scalability, and portability—whether you’re working on a laptop, HPC cluster, or in the cloud. Nextflow workflows are written using DSL2, allowing modular code design and seamless integration with container technologies like Docker, Singularity, Conda, or manual installations.
+[[https://www.nextflow.io/docs/latest/index.html|Official documentation is available here]].
-Submit a ticket:
+=== Running Nextflow on the CHPC Cluster ===
-👉 https://users.chpc.ac.za/helpdesk/tickets/submit/
+CHPC supports Nextflow workflows through Singularity containers. Since compute nodes have no internet access, all dependencies must be downloaded in advance on the login node.
-**Include:**
+=== 1. Connect to the Login Node ===
+Log into the CHPC login node using your Lengau credentials:
+<code bash> ssh username@scp.chpc.ac.za </code>
+Use this session to prepare your workflow and submit jobs.
+=== 2. Load the Nextflow Module ===
+Load the necessary environment modules:
+<code bash>module load chpc/BIOMODULES nextflow
+module load chpc/singularity
+</code>
+Note: Modules must be reloaded in every new session unless added to your ~/.bashrc.
+=== 3. Pull Workflow and Dependencies ===
+Pull your workflow and dependencies on the login1 node:
+Pull the workflow:
+<code bash> nextflow pull nf-core/rnaseq </code>
+Run a test execution:
+<code bash> nextflow run nf-core/rnaseq -profile test </code>
+This will:
+Cache the workflow in ~/.nextflow/assets/
+Download containers (if configured)
+Retrieve auxiliary files and dependencies
+=== Cached Files and Workflow Structure ===
+Workflow code is stored in:
+<code>~/.nextflow/assets</code>
+Container images are stored in:
+<code>~/.singularity</code>
+**🗂 Finding the nextflow.config File**
+After pulling a workflow, you’ll typically find the nextflow.config file in its root directory.
+Example:
+<code bash>
+cd ~/.nextflow/assets/nf-core/rnaseq/
+ls
+</code>
+Look for:
+<code>nextflow.config</code>
+If missing, config files may reside in the conf/ directory or be fetched remotely. You can always override settings by creating your own nextflow.config.
+🚫 No Manual PBS Scripts Needed
+Nextflow automatically generates and submits PBS scripts. You only define resources in nextflow.config.
+=== ⚙️ Configuration with nextflow.config ===
+== 🔧 1. Global Resource Settings ==
+Set default resource usage for all workflow processes:
+<code nextflow> process {
+    executor = 'pbs'
+    withLabel: big_job {
+        queue = 'smp'
+        cpus = 24
+        memory = '120 GB'
+        time = '24h'
+    }
+}
+ </code>
+== 🏷️ 2. Custom Resource Labels ==
+Customize resources for specific process groups using labels:
+<code nextflow>
+process {
+    executor = 'pbs'
+    withLabel: big_job {
+        cpus = 16
+        memory = '64 GB'
+        time = '12h'
+        queue = 'smp'
+    }
+    withLabel: short_job {
+        cpus = 1
+        memory = '1 GB'
+        time = '15m'
+        queue = 'smp'
+    }
+}
+</code>
+Use the label in your pipeline:
+<code nextflow>
+process bigTask {
+label 'big_job'
+...
+}
+</code>
+== 📦 3. Singularity Integration ==
+Enable Singularity support:
+<code nextflow> singularity.enabled = true
+singularity.autoMounts = true </code>
+Specify containers:
+From Docker Hub:
+<code nextflow>
+process.container = 'docker://biocontainers/fastqc:v0.11.9_cv8'
+</code>
+From local image:
+<code nextflow>
+process.container = '/path/to/image.sif'
+</code>
+Set cache directory to avoid re-downloads:
+<code nextflow> singularity.cacheDir = '/path/to/.singularity' </code>
+== 🖥️ 4. PBS Executor Settings ==
+Customize PBS job submission:
+<code nextflow>
+executor {
+  name = 'pbs'
+  queueSize = 20
+  submitOptions = '-V -m abe -M your@email.com'
+}
+</code>
+== 📂 5. Using Profiles ==
+Profiles let you switch configurations easily:
+<code nextflow> profiles {
+  standard {
+    process.executor = 'pbs'
+    process.queue = 'smp'
+  }
+  local {
+    process.executor = 'local'
+    docker.enabled = false
+  }
+  cluster_singularity {
+    process.executor = 'pbs'
+    singularity.enabled = true
+    process.container = 'file:///path/to/container.sif'
+  }
+}
+ </code>
+Run a profile:
+<code bash> nextflow run main.nf -profile cluster_singularity </code>
+== 🚫 Offline Mode ==
+Run jobs on compute nodes without internet access:
+<code bash> nextflow run ~/.nextflow/assets/nf-core/rnaseq \
+-profile singularity -offline </code>
+⚠️ Always include -offline on compute nodes to prevent online fetching.
+=== 🧭 Debugging and Logs ===
+Each Nextflow process generates a unique work directory (work/ab/xyz123), containing:
+.command.run — generated PBS job script
+.command.sh — wrapped shell script
+.command.log — job output
+.exitcode — exit status
+To inspect a failed job:
+<code bash> cd work/ab/xyz123/
+less .command.log </code>
+=== ✅ Summary ===
+nextflow.config centralizes all pipeline settings.
+No need to write PBS scripts manually.
+Resources, container usage, and submission options are all configurable.
+Profiles improve portability and reproducibility.
+Offline mode is essential for CHPC compute node compatibility.
+🧠 Tip: For workflows requiring reference data, bind directories just like with containers:
+<code bash> nextflow run /path/to/my_pipeline -profile singularity -offline \
+--input /data/input.fastq \
+--genomeDir /mnt/lustre/bsp/DB/genomes
+</code>
+🧹 Clean Up:
+Nextflow stores all its cache files in your home directory, so it's important to clean up these files once you're finished using a workflow to avoid running out of space.
+<code bash>
+rm -rf ~/.nextflow/assets/
+rm -rf ~/.nextflow/tmp
+rm -rf ~/.singularity
+</code>
-The commands you ran, singularity image used, paths and error logs
+=== Need Help? ===
+If you encounter issues or need a specific tool installed contact the CHPC support team at:
+  * 📧 help@chpc.ac.za
+  * https://users.chpc.ac.za/helpdesk/tickets/submit/
+Include your job script and all errors encountered.
 ===== Basic examples =====

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools