wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » mfix
Trace:
howto:mfix

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:mfix [2023/08/17 10:08]
ccrosby 		howto:mfix [2025/04/16 12:28] (current)
ccrosby [MFix-Exa]
Line 1: Line 1:
 ======Using MFIX at the CHPC====== ======Using MFIX at the CHPC======
 +Instructional video: [[https://youtu.be/3ewsQg2WD3o|MFix at the CHPC]].
 ===== Introduction ===== ===== Introduction =====
 MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC. MFIX is an open-source multiphase flow solver written in FORTRAN 90. It is used for simulating fluid-solid systems such as fluidized beds. As it appears to have numerous potential uses in chemical engineering and mineral processing applications, a limited level of support for the use of the package is now available at the CHPC.
Line 15: Line 16:
   - Through a very slick Python-based GUI.   - Through a very slick Python-based GUI.
   - By customizing the solver and running from the command line.   - By customizing the solver and running from the command line.
-However, even when using the customized command line approach, a Python script is still used for building the code.  The following instructions apply to MFix-23.2, which has been set up to facilitate both of these approaches.+However, even when using the customized command line approach, a Python script is still used for building the code.  The following instructions apply from MFix-23.2 onwards, which have been set up to facilitate both of these approaches.  
  
-==== MFix-23.2 =====+==== MFix-Exa ==== 
 +Further to the theme of modernised MFix, is the development of [[https://mfix.netl.doe.gov/products/mfix-exa/|MFix-Exa]], a high-performance, massively parallel, machine agnostic multiphase flow code for the exascale computing era. MFIX-Exa is built using the AMReX framework which provides iterators, linear solvers, parallel communication routines, and other utilities that support highly efficient operations on structured grid and particle data. MFIX-Exa performance portability is inherited from AMReX’s flexible MPI+X hybrid parallelism strategy allowing MFIX-Exa to offload to Nvidia, AMD, and Intel GPU accelerators.  Due to a scandalous shortage of GPUs, the GPU version is not available at the CHPC.  The latest installed version is 2025-04.1, installed in the directory ''/home/apps/chpc/compmech/mfix-exa-25.04.1''  The executable is installed in the subdirectory ''exec''.  MFix-exa-25.04.1 on Lengau has been compiled with Intel's OneAPI-2023.2.0 compiler and mpich-4.2.2 with support for OpenMP.  The "superbuild" executable is called ''mfix3d.intel-llvm.MPI.OMP.ex'', but is also accessible with a symbolic link ''mfix-exa''.  To set up the environment, source the following script: 
 +<code>.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel </code> 
 + 
 +=== Example job script for MFix-Exa === 
 +<file bash runMFix-exa.pbs> 
 +#!/bin/bash 
 +### Request two compute nodes, each with 12 MPI processes and 2 OpenMP threads per MPI process 
 +#PBS -l select=2:ncpus=24:mpiprocs=12 
 +#PBS -P MECH1234 
 +#PBS -l walltime=02:00:00 
 +#PBS -q normal 
 +#PBS -o /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stdout 
 +#PBS -e /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03/mfix.stderr 
 + 
 +### Change directory to a typical test case that comes with MFix.  Obviously use your own directory.  
 +cd /home/jblogs/lustre/mfix-exa/tests/fluid/FLD03 
 + 
 +### Set up the required environment 
 +.  /home/apps/chpc/compmech/mfix-exa-25.04.1/setMfixIntel 
 + 
 +### Set up 2 OpenMP threads per MPI process 
 +export OMP_NUM_THREADS=2 
 + 
 +### Find the number of MPI processes 
 +nproc=`cat $PBS_NODEFILE | wc -l` 
 + 
 +mpirun -iface ib0 -np $nproc mfix-exa inputs > mfix.out 
 +</file> 
 + 
 + 
 + 
 +==== MFix-24.2.3 ==== 
 +The latest installation makes use of Miniforge-3.  To set up a suitable environment (starting from a clean base), execute the following steps: 
 + 
 +<code> 
 +module load chpc/compmech/python/miniforge-3 
 +conda init 
 +conda activate mfix-24.2.3 
 +</code> 
 + 
 +You will notice that mfix has been installed in the directory as shown below: 
 + 
 +<code> 
 +(base) [jblogs@cnode1234:~]$ conda activate mfix-24.2.3 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ which mfix 
 +/home/apps/chpc/compmech/MFix/miniforge3/envs/mfix-24.2.3/bin/mfix 
 +(mfix-24.2.3) [jblogs@cnode1234:~]$ 
 +</code> 
 + 
 +To exit this environment, use the command: 
 +<code> 
 +conda deactivate 
 +</code> 
 + 
 +The MFix source code and tutorials have been unpacked in the directory  
 +<code> /home/apps/chpc/compmech/MFix/mfix-24.2.3 </code> 
 + 
 + 
 +==== MFix-23.2 ====
 A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/home/apps/chpc/compmech/MFix/miniconda3/envs/mfix-23.2''.  These are the required steps: A Miniconda environment for MFix-23.2 has been configured, and is installed in the directory ''/home/apps/chpc/compmech/MFix/miniconda3/envs/mfix-23.2''.  These are the required steps:
  
Line 51: Line 111:
 </code> </code>
  
 +=== Example job script ===
 +<file bash runMFix.pbs>
 +#!/bin/bash
 +###
 +###
 +### The same workflow should work for 23.2 and 23.3, just substitute 23.2 with 23.3 if 
 +###  you need the more recent version.
 +###
 +###
 +### Request a single node for 4 MPI processes and 2 OpenMP threads per MPI process
 +#PBS -l select=1:ncpus=8:mpiprocs=4
 +#PBS -P MECH1234
 +#PBS -l walltime=02:00:00
 +#PBS -q serial
 +#PBS -o /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03/mfix.out
 +#PBS -e /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03/mfix.err
  
 +### Change directory to a typical test case that comes with MFix.  Obviously use your own directory. 
 +cd /home/jblogs/lustre/mfix-23.2/tests/fluid/FLD03
 +
 +### Prepare the Miniconda environment
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh
 +
 +### Get the following into your path:
 +###   1. The standard MFix installation
 +###   2. gcc-8.3.0 (not used for 23.3)
 +###   3. mpich-3.3 (not used for 23.3)
 +module load chpc/compmech/mfix/23.2
 +
 +### Activate the mfix conda environment
 +conda activate mfix-23.2
 +
 +### Build your custom solver for both OpenMP and MPI.  Use 4 threads for the compile.
 +build_mfixsolver --batch --smp --dmp -j 4
 +
 +### We no longer need the mfix conda environment.  Deactivate it.
 +conda deactivate
 +
 +### Set up 2 OpenMP threads per MPI process
 +export OMP_NUM_THREADS=2
 +
 +### The MFix test case comes with a script to run the case.  This script contains the mpirun command which 
 +###  can be edited.  In this particular case it had te be edited to remove two options pertaining to 
 +###  running as root and oversubscribing cores.  The mpirun command looks like this:
 +###   mpirun -np 4 ./mfixsolver -f mfix.dat nodesi=2 nodesj=2  
 +###  and can also be used below instead of the the script.
 +
 +./runtests.sh | tee mfixsolver.out
 +
 +</file>
 +
 +==== Interactive MFix ====
 +With the environment set up by these instructions: 
 +<code> 
 +source /home/apps/chpc/compmech/MFix/miniconda3/etc/profile.d/conda.sh
 +module load chpc/compmech/mfix/23.2
 +conda activate mfix-23.2
 +</code>
 +it is also possible to run the interactive version of MFix with its Python-based GUI.  However, there are two main problems:
 +  - Doing graphics with X-forwarding is slow 
 +  - Compute nodes do not have GPUs that can handle OpenGL graphics
  
-===== Building MFIX =====+These two problems are easy to overcome.  Use VNC to work with a virtual desktop and parallel Mesa software rendering to provide OpenGL graphics.  Please read this page: https://wiki.chpc.ac.za/howto:remote_viz 
 +===== Working with older versions ===== 
 +==== Building MFIX ====
 MFIX generates a new executable for each case, which is copied in to the case directory. MFIX generates a new executable for each case, which is copied in to the case directory.
 First cd to the case directory. First cd to the case directory.
Line 78: Line 200:
 It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations). It should be noted that use of the Johnson and Jackson partial slip boundary condition (BC_JJ in mfix.dat file) causes a crash for all methods of parallelisation (although it works for serial computations).
  
-===== Setting up and submitting a job =====+==== Setting up and submitting a job ====
  
 The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp. The input for an MFIX case consists of an mfix.dat file, a text file which defines most or all of the properties of case (geometry, boundary and initial conditions, choice of turbulence and friction models, and so on), as well as any fortran source files containing user-extensions to the standard MFIX solver, and any additional optional files describing geometry. In most cases the mfix.dat file is sufficient, and as this file is relatively small, it may be uploaded using scp.
Line 90: Line 212:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /mnt/lustre3p/users/username/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /mnt/lustre3p/users/username/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
Line 116: Line 238:
   #PBS -q normal   #PBS -q normal
   #PBS -l walltime=01:00:00   #PBS -l walltime=01:00:00
-  #PBS -o /mnt/lustre3p/users/username/job01/stdout +  #PBS -o /mnt/lustre/users/username/job01/stdout 
-  #PBS -e /mnt/lustre3p/users/username/job01/stderr+  #PBS -e /mnt/lustre/users/username/job01/stderr
   #PBS -m abe   #PBS -m abe
   #PBS -M username@email.co.za   #PBS -M username@email.co.za
/app/dokuwiki/data/attic/howto/mfix.1692259721.txt.gz · Last modified: 2023/08/17 10:08 by ccrosby

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki