howto:mfix
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
howto:mfix [2022/09/26 16:25] ccrosby |
howto:mfix [2025/04/16 12:28] (current) ccrosby [MFix-Exa] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ======Using MFIX at the CHPC====== | ======Using MFIX at the CHPC====== | ||
| + | Instructional video: [[https:// | ||
| ===== Introduction ===== | ===== Introduction ===== | ||
| - | MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, | + | MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, |
| Alternatively, | Alternatively, | ||
| Line 11: | Line 12: | ||
| For a more general explanation of the use of MFIX, consult the documentation and example cases included with the source tarball. | For a more general explanation of the use of MFIX, consult the documentation and example cases included with the source tarball. | ||
| + | ===== Modern MFix ===== | ||
| + | In recent years, MFix has been substantially modernised. | ||
| + | - Through a very slick Python-based GUI. | ||
| + | - By customizing the solver and running from the command line. | ||
| + | However, even when using the customized command line approach, a Python script is still used for building the code. The following instructions apply from MFix-23.2 onwards, which have been set up to facilitate both of these approaches. | ||
| - | ===== Building | + | ==== MFix-Exa |
| + | Further to the theme of modernised MFix, is the development of [[https:// | ||
| + | < | ||
| + | |||
| + | === Example job script for MFix-Exa === | ||
| + | <file bash runMFix-exa.pbs> | ||
| + | # | ||
| + | ### Request two compute nodes, each with 12 MPI processes and 2 OpenMP threads per MPI process | ||
| + | #PBS -l select=2: | ||
| + | #PBS -P MECH1234 | ||
| + | #PBS -l walltime=02: | ||
| + | #PBS -q normal | ||
| + | #PBS -o / | ||
| + | #PBS -e / | ||
| + | |||
| + | ### Change directory to a typical test case that comes with MFix. Obviously use your own directory. | ||
| + | cd / | ||
| + | |||
| + | ### Set up the required environment | ||
| + | . / | ||
| + | |||
| + | ### Set up 2 OpenMP threads per MPI process | ||
| + | export OMP_NUM_THREADS=2 | ||
| + | |||
| + | ### Find the number of MPI processes | ||
| + | nproc=`cat $PBS_NODEFILE | wc -l` | ||
| + | |||
| + | mpirun -iface ib0 -np $nproc mfix-exa inputs > mfix.out | ||
| + | </ | ||
| + | |||
| + | |||
| + | |||
| + | ==== MFix-24.2.3 ==== | ||
| + | The latest installation makes use of Miniforge-3. | ||
| + | |||
| + | < | ||
| + | module load chpc/ | ||
| + | conda init | ||
| + | conda activate mfix-24.2.3 | ||
| + | </ | ||
| + | |||
| + | You will notice that mfix has been installed in the directory as shown below: | ||
| + | |||
| + | < | ||
| + | (base) [jblogs@cnode1234: | ||
| + | (mfix-24.2.3) [jblogs@cnode1234: | ||
| + | / | ||
| + | (mfix-24.2.3) [jblogs@cnode1234: | ||
| + | </ | ||
| + | |||
| + | To exit this environment, | ||
| + | < | ||
| + | conda deactivate | ||
| + | </ | ||
| + | |||
| + | The MFix source code and tutorials have been unpacked in the directory | ||
| + | < | ||
| + | |||
| + | |||
| + | ==== MFix-23.2 ==== | ||
| + | A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/ | ||
| + | |||
| + | === Step 1: Source the conda shell script === | ||
| + | This command can be issued from the command line or be placed in your '' | ||
| + | < | ||
| + | source / | ||
| + | </ | ||
| + | |||
| + | === Step 2: Load the appropriate mfix module === | ||
| + | Like the previous instruction, | ||
| + | < | ||
| + | module load chpc/ | ||
| + | </ | ||
| + | |||
| + | === Step 3: Activate the conda environment === | ||
| + | This step is necessary if you want to use the GUI-based process or if you want to compile a custom solver. | ||
| + | < | ||
| + | conda activate mfix-23.2 | ||
| + | </ | ||
| + | |||
| + | === Step 4: Compile a custom solver === | ||
| + | The standard '' | ||
| + | < | ||
| + | build_mfixsolver --batch --smp --dmp -j 4 | ||
| + | </ | ||
| + | |||
| + | === Step 5: Run the solver === | ||
| + | If you are going to run a custom solver, first remove old out files, then set your required number of OpenMP threads, if appropriate, | ||
| + | < | ||
| + | export OMP_NUM_THREADS=4 | ||
| + | ./ | ||
| + | </ | ||
| + | |||
| + | === Example job script === | ||
| + | <file bash runMFix.pbs> | ||
| + | # | ||
| + | ### | ||
| + | ### | ||
| + | ### The same workflow should work for 23.2 and 23.3, just substitute 23.2 with 23.3 if | ||
| + | ### you need the more recent version. | ||
| + | ### | ||
| + | ### | ||
| + | ### Request a single node for 4 MPI processes and 2 OpenMP threads per MPI process | ||
| + | #PBS -l select=1: | ||
| + | #PBS -P MECH1234 | ||
| + | #PBS -l walltime=02: | ||
| + | #PBS -q serial | ||
| + | #PBS -o / | ||
| + | #PBS -e / | ||
| + | |||
| + | ### Change directory to a typical test case that comes with MFix. Obviously use your own directory. | ||
| + | cd / | ||
| + | |||
| + | ### Prepare the Miniconda environment | ||
| + | source / | ||
| + | |||
| + | ### Get the following into your path: | ||
| + | ### 1. The standard MFix installation | ||
| + | ### 2. gcc-8.3.0 (not used for 23.3) | ||
| + | ### 3. mpich-3.3 (not used for 23.3) | ||
| + | module load chpc/ | ||
| + | |||
| + | ### Activate the mfix conda environment | ||
| + | conda activate mfix-23.2 | ||
| + | |||
| + | ### Build your custom solver for both OpenMP and MPI. Use 4 threads for the compile. | ||
| + | build_mfixsolver --batch --smp --dmp -j 4 | ||
| + | |||
| + | ### We no longer need the mfix conda environment. | ||
| + | conda deactivate | ||
| + | |||
| + | ### Set up 2 OpenMP threads per MPI process | ||
| + | export OMP_NUM_THREADS=2 | ||
| + | |||
| + | ### The MFix test case comes with a script to run the case. This script contains the mpirun command which | ||
| + | ### can be edited. | ||
| + | ### running as root and oversubscribing cores. | ||
| + | ### | ||
| + | ### and can also be used below instead of the the script. | ||
| + | |||
| + | ./ | ||
| + | |||
| + | </ | ||
| + | |||
| + | ==== Interactive MFix ==== | ||
| + | With the environment set up by these instructions: | ||
| + | < | ||
| + | source / | ||
| + | module load chpc/ | ||
| + | conda activate mfix-23.2 | ||
| + | </ | ||
| + | it is also possible to run the interactive version of MFix with its Python-based GUI. However, there are two main problems: | ||
| + | - Doing graphics with X-forwarding is slow | ||
| + | - Compute nodes do not have GPUs that can handle OpenGL graphics | ||
| + | |||
| + | These two problems are easy to overcome. | ||
| + | ===== Working with older versions ===== | ||
| + | ==== Building MFIX ==== | ||
| MFIX generates a new executable for each case, which is copied in to the case directory. | MFIX generates a new executable for each case, which is copied in to the case directory. | ||
| First cd to the case directory. | First cd to the case directory. | ||
| Line 37: | Line 200: | ||
| It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations). | It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations). | ||
| - | ===== Setting up and submitting a job ===== | + | ==== Setting up and submitting a job ==== |
| The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp. | The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp. | ||
| Line 49: | Line 212: | ||
| #PBS -q normal | #PBS -q normal | ||
| #PBS -l walltime=01: | #PBS -l walltime=01: | ||
| - | #PBS -o /mnt/lustre3p/ | + | #PBS -o /mnt/lustre/ |
| - | #PBS -e /mnt/lustre3p/ | + | #PBS -e /mnt/lustre/ |
| #PBS -m abe | #PBS -m abe | ||
| #PBS -M username@email.co.za | #PBS -M username@email.co.za | ||
| Line 75: | Line 238: | ||
| #PBS -q normal | #PBS -q normal | ||
| #PBS -l walltime=01: | #PBS -l walltime=01: | ||
| - | #PBS -o /mnt/lustre3p/ | + | #PBS -o /mnt/lustre/ |
| - | #PBS -e /mnt/lustre3p/ | + | #PBS -e /mnt/lustre/ |
| #PBS -m abe | #PBS -m abe | ||
| #PBS -M username@email.co.za | #PBS -M username@email.co.za | ||
/app/dokuwiki/data/attic/howto/mfix.1664202307.txt.gz · Last modified: 2022/09/26 16:25 by ccrosby
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
