wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » mfix
Trace:
howto:mfix

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:mfix [2023/08/17 13:18]
ccrosby [MFix-23.2] 		howto:mfix [2025/04/16 12:28] (current)
ccrosby [MFix-Exa]
Line 1: Line 1:
 ======Using MFIX at the CHPC====== ======Using MFIX at the CHPC======
 +Instructional video: [[https://youtu.be/3ewsQg2WD3o|MFix at the CHPC]].
 ===== Introduction ===== ===== Introduction =====
 MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC. MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC.
Line 15: Line 16:
   - Through a very slick Python-based GUI.   - Through a very slick Python-based GUI.
   - By customizing the solver and running from the command line.   - By customizing the solver and running from the command line.
-However, even when using the customized command line approach, a Python script is still used for building the code.  The following instructions apply to MFix-23.2, which has been set up to facilitate both of these approaches.+However, even when using the customized command line approach, a Python script is still used for building the code.  The following instructions apply from MFix-23.2 onwards, which have been set up to facilitate both of these approaches.  
  
-==== MFix-23.2 =====+==== MFix-Exa ==== 
 +Further to the theme of modernised MFix, is the development of [[https://mfix.netl.doe.gov/products/mfix-exa/|MFix-Exa]], a high-performance, massively parallel, machine agnostic multiphase flow code for the exascale computing era. MFIX-Exa is built using the AMReX framework which provides iterators, linear solvers, parallel communication routines, and other utilities that support highly efficient operations on structured grid and particle data. MFIX-Exa performance portability is inherited from AMReX’s flexible MPI+X hybrid parallelism strategy allowing MFIX-Exa to offload to Nvidia, AMD, and Intel GPU accelerators.  Due to a scandalous shortage of GPUs, the GPU version is not available at the CHPC.  The latest installed version is 2025-04.1, installed in the directory ''/home/apps/chpc/compmech/mfix-exa-25.04.1''  The executable is installed in the subdirectory ''exec''.  MFix-exa-25.04.1 on Lengau has been compiled with Intel's OneAPI-2023.2.0 compiler and mpich-4.2.2 with support for OpenMP.  The "superbuild" executable is called ''mfix3d.intel-llvm.MPI.OMP.ex'', but is also accessible with a symbolic link ''mfix-exa''.  To set up the environment, source the following script: 
 +<code>.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel </code> 
 + 
 +=== Example job script for MFix-Exa === 
 +<file bash runMFix-exa.pbs> 
 +#!/bin/bash 
 +### Request two compute nodes, each with 12 MPI processes and 2 OpenMP threads per MPI process 
 +#PBS -l select=2:ncpus=24:mpiprocs=12 
 +#PBS -P MECH1234 
 +#PBS -l walltime=02:00:00 
 +#PBS -q normal 
 +#PBS -o /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stdout 
 +#PBS -e /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stderr 
 + 
 +### Change directory to a typical test case that comes with MFix.  Obviously use your own directory.  
 +cd /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03 
 + 
 +### Set up the required environment 
 +.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel 
 + 
 +### Set up 2 OpenMP threads per MPI process 
 +export OMP_NUM_THREADS=2 
 + 
 +### Find the number of MPI processes 
 +nproc=`cat $PBS_NODEFILE | wc -l` 
 + 
 +mpirun -iface ib0 -np $nproc mfix-exa inputs > mfix.out 
 +</file> 
 + 
 + 
 + 
 +==== MFix-24.2.3 ==== 
 +The latest installation makes use of Miniforge-3.  To set up a suitable environment (starting from a clean base), execute the following steps: 
 + 
 +<code> 
 +module load chpc/compmech/python/miniforge-3 
 +conda init 
 +conda activate mfix-24.2.3 
 +</code> 
 + 
 +You will notice that mfix has been installed in the directory as shown below: 
 + 
 +<code> 
 +(base) [jblogs@cnode1234:~]$ conda activate mfix-24.2.3 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ which mfix 
 +/home/apps/chpc/compmech/MFix/miniforge3/envs/mfix-24.2.3/bin/mfix 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ 
 +</code> 
 + 
 +To exit this environment, use the command: 
 +<code> 
 +conda deactivate 
 +</code> 
 + 
 +The MFix source code and tutorials have been unpacked in the directory  
 +<code> /home/apps/chpc/compmech/MFix/mfix-24.2.3 </code> 
 + 
 + 
 +==== MFix-23.2 ====
 A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/home/apps/chpc/compmech/MFix/miniconda3/envs/mfix-23.2''.  These are the required steps: A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/home/apps/chpc/compmech/MFix/miniconda3/envs/mfix-23.2''.  These are the required steps:
  
Line 54: Line 114:
 <file bash runMFix.pbs> <file bash runMFix.pbs>
 #!/bin/bash #!/bin/bash
 +###
 +###
 +### The same workflow should work for 23.2 and 23.3, just substitute 23.2 with 23.3 if 
 +###  you need the more recent version.
 +###
 +###
 ### Request a single node for 4 MPI processes and 2 OpenMP threads per MPI process ### Request a single node for 4 MPI processes and 2 OpenMP threads per MPI process
 #PBS -l select=1:ncpus=8:mpiprocs=4 #PBS -l select=1:ncpus=8:mpiprocs=4
Line 70: Line 136:
 ### Get the following into your path: ### Get the following into your path:
 ###   1. The standard MFix installation ###   1. The standard MFix installation
-###   2. gcc-8.3.0 +###   2. gcc-8.3.0 (not used for 23.3) 
-###   3. mpich-3.3+###   3. mpich-3.3 (not used for 23.3)
 module load chpc/compmech/mfix/23.2 module load chpc/compmech/mfix/23.2
  
Line 96: Line 162:
 </file> </file>
  
 +==== Interactive MFix ====
 +With the environment set up by these instructions: 
 +<code> 
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh
 +module load chpc/compmech/mfix/23.2
 +conda activate mfix-23.2
 +</code>
 +it is also possible to run the interactive version of MFix with its Python-based GUI.  However, there are two main problems:
 +  - Doing graphics with X-forwarding is slow 
 +  - Compute nodes do not have GPUs that can handle OpenGL graphics
  
 +These two problems are easy to overcome.  Use VNC to work with a virtual desktop and parallel Mesa software rendering to provide OpenGL graphics.  Please read this page: https://wiki.chpc.ac.za/howto:remote_viz
 ===== Working with older versions ===== ===== Working with older versions =====
 ==== Building MFIX ==== ==== Building MFIX ====
Line 135: Line 212:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /mnt/lustre3p/users/username/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /mnt/lustre3p/users/username/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
Line 161: Line 238:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /mnt/lustre3p/users/username/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /mnt/lustre3p/users/username/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
/app/dokuwiki/data/attic/howto/mfix.1692271090.txt.gz · Last modified: 2023/08/17 13:18 by ccrosby

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki