wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » mfix
Trace:
howto:mfix

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:mfix [2021/05/05 11:45]
ccrosby [Setting up and submitting a job] 		howto:mfix [2025/04/16 12:28] (current)
ccrosby [MFix-Exa]
Line 1: Line 1:
 ======Using MFIX at the CHPC====== ======Using MFIX at the CHPC======
 +Instructional video: [[https://youtu.be/3ewsQg2WD3o|MFix at the CHPC]].
 ===== Introduction ===== ===== Introduction =====
-MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC, at /apps/chpc/compmech/CFD/MFIX .+MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC.
  
 Alternatively, users may download the code and install it in their home directory. Alternatively, users may download the code and install it in their home directory.
Line 11: Line 12:
 For a more general explanation of the use of MFIX, consult the documentation and example cases included with the source tarball. For a more general explanation of the use of MFIX, consult the documentation and example cases included with the source tarball.
  
 +===== Modern MFix =====
 +In recent years, MFix has been substantially modernised.  It can be used in two ways:
 +  - Through a very slick Python-based GUI.
 +  - By customizing the solver and running from the command line.
 +However, even when using the customized command line approach, a Python script is still used for building the code.  The following instructions apply from MFix-23.2 onwards, which have been set up to facilitate both of these approaches.  
  
-===== Building MFIX =====+==== MFix-Exa ==== 
 +Further to the theme of modernised MFix, is the development of [[https://mfix.netl.doe.gov/products/mfix-exa/|MFix-Exa]], a high-performance, massively parallel, machine agnostic multiphase flow code for the exascale computing era. MFIX-Exa is built using the AMReX framework which provides iterators, linear solvers, parallel communication routines, and other utilities that support highly efficient operations on structured grid and particle data. MFIX-Exa performance portability is inherited from AMReX’s flexible MPI+X hybrid parallelism strategy allowing MFIX-Exa to offload to Nvidia, AMD, and Intel GPU accelerators.  Due to a scandalous shortage of GPUs, the GPU version is not available at the CHPC.  The latest installed version is 2025-04.1, installed in the directory ''/home/apps/chpc/compmech/mfix-exa-25.04.1''  The executable is installed in the subdirectory ''exec''.  MFix-exa-25.04.1 on Lengau has been compiled with Intel's OneAPI-2023.2.0 compiler and mpich-4.2.2 with support for OpenMP.  The "superbuild" executable is called ''mfix3d.intel-llvm.MPI.OMP.ex'', but is also accessible with a symbolic link ''mfix-exa''.  To set up the environment, source the following script: 
 +<code>.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel </code> 
 + 
 +=== Example job script for MFix-Exa === 
 +<file bash runMFix-exa.pbs> 
 +#!/bin/bash 
 +### Request two compute nodes, each with 12 MPI processes and 2 OpenMP threads per MPI process 
 +#PBS -l select=2:ncpus=24:mpiprocs=12 
 +#PBS -P MECH1234 
 +#PBS -l walltime=02:00:00 
 +#PBS -q normal 
 +#PBS -o /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stdout 
 +#PBS -e /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stderr 
 + 
 +### Change directory to a typical test case that comes with MFix.  Obviously use your own directory.  
 +cd /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03 
 + 
 +### Set up the required environment 
 +.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel 
 + 
 +### Set up 2 OpenMP threads per MPI process 
 +export OMP_NUM_THREADS=2 
 + 
 +### Find the number of MPI processes 
 +nproc=`cat $PBS_NODEFILE | wc -l` 
 + 
 +mpirun -iface ib0 -np $nproc mfix-exa inputs > mfix.out 
 +</file> 
 + 
 + 
 + 
 +==== MFix-24.2.3 ==== 
 +The latest installation makes use of Miniforge-3.  To set up a suitable environment (starting from a clean base), execute the following steps: 
 + 
 +<code> 
 +module load chpc/compmech/python/miniforge-3 
 +conda init 
 +conda activate mfix-24.2.3 
 +</code> 
 + 
 +You will notice that mfix has been installed in the directory as shown below: 
 + 
 +<code> 
 +(base) [jblogs@cnode1234:~]$ conda activate mfix-24.2.3 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ which mfix 
 +/home/apps/chpc/compmech/MFix/miniforge3/envs/mfix-24.2.3/bin/mfix 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ 
 +</code> 
 + 
 +To exit this environment, use the command: 
 +<code> 
 +conda deactivate 
 +</code> 
 + 
 +The MFix source code and tutorials have been unpacked in the directory  
 +<code> /home/apps/chpc/compmech/MFix/mfix-24.2.3 </code> 
 + 
 + 
 +==== MFix-23.2 ==== 
 +A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/home/apps/chpc/compmech/MFix/miniconda3/envs/mfix-23.2''.  These are the required steps: 
 + 
 +=== Step 1: Source the conda shell script === 
 +This command can be issued from the command line or be placed in your ''$HOME/.bashrc'' file for interactive work.  When running in non-interactive batch mode through PBS, it should be placed in your PBS script.  
 +<code> 
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh 
 +</code>  
 + 
 +=== Step 2: Load the appropriate mfix module === 
 +Like the previous instruction, this can be given from the command line or placed in your ''$HOME/.bashrc'' file for interactive work, or placed in the PBS job script. 
 +<code> 
 +module load chpc/compmech/mfix/23.2 
 +</code> 
 + 
 +=== Step 3: Activate the conda environment === 
 +This step is necessary if you want to use the GUI-based process or if you want to compile a custom solver.  It is not necessary if you simply want to run an already-compiled custom solver. 
 +<code> 
 +conda activate mfix-23.2 
 +</code> 
 + 
 +=== Step 4: Compile a custom solver === 
 +The standard ''mfixsolver'' is already in your path.  To build a custom solver that incorporates user coding and/or parallelisation, use the provided ''build_mfixsolver'' procedure with the appropriate options.  This example provides both smp (OpenMP) and dmp (MPI) parallelisation and is compiled using 4 threads. 
 +<code> 
 +build_mfixsolver --batch --smp --dmp -j 4 
 +</code> 
 + 
 +=== Step 5: Run the solver === 
 +If you are going to run a custom solver, first remove old out files, then set your required number of OpenMP threads, if appropriate, and run.  This example is for smp only. 
 +<code> 
 +export OMP_NUM_THREADS=4 
 +./mfixsolver 
 +</code> 
 + 
 +=== Example job script === 
 +<file bash runMFix.pbs> 
 +#!/bin/bash 
 +### 
 +### 
 +### The same workflow should work for 23.2 and 23.3, just substitute 23.2 with 23.3 if  
 +###  you need the more recent version. 
 +### 
 +### 
 +### Request a single node for 4 MPI processes and 2 OpenMP threads per MPI process 
 +#PBS -l select=1:ncpus=8:mpiprocs=4 
 +#PBS -P MECH1234 
 +#PBS -l walltime=02:00:00 
 +#PBS -q serial 
 +#PBS -o /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03/mfix.out 
 +#PBS -e /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03/mfix.err 
 + 
 +### Change directory to a typical test case that comes with MFix.  Obviously use your own directory.  
 +cd /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03 
 + 
 +### Prepare the Miniconda environment 
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh 
 + 
 +### Get the following into your path: 
 +###   1. The standard MFix installation 
 +###   2. gcc-8.3.0 (not used for 23.3) 
 +###   3. mpich-3.3 (not used for 23.3) 
 +module load chpc/compmech/mfix/23.2 
 + 
 +### Activate the mfix conda environment 
 +conda activate mfix-23.2 
 + 
 +### Build your custom solver for both OpenMP and MPI.  Use 4 threads for the compile. 
 +build_mfixsolver --batch --smp --dmp -j 4 
 + 
 +### We no longer need the mfix conda environment.  Deactivate it. 
 +conda deactivate 
 + 
 +### Set up 2 OpenMP threads per MPI process 
 +export OMP_NUM_THREADS=2 
 + 
 +### The MFix test case comes with a script to run the case.  This script contains the mpirun command which  
 +###  can be edited.  In this particular case it had te be edited to remove two options pertaining to  
 +###  running as root and oversubscribing cores.  The mpirun command looks like this: 
 +###   mpirun -np 4 ./mfixsolver -f mfix.dat nodesi=2 nodesj=2   
 +###  and can also be used below instead of the the script. 
 + 
 +./runtests.sh | tee mfixsolver.out 
 + 
 +</file> 
 + 
 +==== Interactive MFix ==== 
 +With the environment set up by these instructions:  
 +<code>  
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh 
 +module load chpc/compmech/mfix/23.2 
 +conda activate mfix-23.2 
 +</code> 
 +it is also possible to run the interactive version of MFix with its Python-based GUI.  However, there are two main problems: 
 +  - Doing graphics with X-forwarding is slow  
 +  - Compute nodes do not have GPUs that can handle OpenGL graphics 
 + 
 +These two problems are easy to overcome.  Use VNC to work with a virtual desktop and parallel Mesa software rendering to provide OpenGL graphics.  Please read this page: https://wiki.chpc.ac.za/howto:remote_viz 
 +===== Working with older versions ===== 
 +==== Building MFIX ====
 MFIX generates a new executable for each case, which is copied in to the case directory. MFIX generates a new executable for each case, which is copied in to the case directory.
 First cd to the case directory. First cd to the case directory.
Line 37: Line 200:
 It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations). It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations).
  
-===== Setting up and submitting a job =====+==== Setting up and submitting a job ====
  
 The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp. The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp.
Line 49: Line 212:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /mnt/lustre3p/users/username/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /mnt/lustre3p/users/username/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
Line 75: Line 238:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /home/username/lustre/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /home/username/lustre/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
Line 85: Line 248:
   mpirun -iface ib0 -np $nproc $exe -parallel >"coarse.log" 2>&1   mpirun -iface ib0 -np $nproc $exe -parallel >"coarse.log" 2>&1
 </file>   </file>  
 +
 +====  MFix-22.2.2 ====
 +Please be advised that there is now also a version 22.2.2 in ''/home/apps/chpc/compmech/MFix-22.2.2''.  The executable ''mfixsolver'' can be added to your path with the following module command:
 +<code>
 +module load chpc/compmech/mfix/22.2.2
 +</code>
 +Please note that this has been compiled with gfortran-9.2.0 and MPICH-4.0.  When running in parallel, add the option ''-iface ib0'' to ensure that the high speed Infiniband network is used.  This version of MPICH has been compiled with support for PBS, therefore a hosts or machine file is not needed.  This command line should work:
 +<code>
 +mpirun -iface ib0 -np $nproc mfixsolver -f myinputfile.mfx
 +</code>
 +
  
 ===== Postprocessing ===== ===== Postprocessing =====
/app/dokuwiki/data/attic/howto/mfix.1620207948.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki