The following instructions are courtesy of Dr. Muaaz Bhamjee of the University of Johannesburg.
Palabos is an open source CFD package based on the Lattice Boltzmann Method (LBM). Palabos is effectively a C++ library that can be used to develop CFD applications. More information can be obtained at the Palabos website.
Refer to the following book for an introduction to the LBM: M. C. Sukop and J. D. T. Thorne, Lattice Boltzmann Modelling: An Introduction for Geoscientists and Engineers, 2nd ed. Miami. FL: Springer-Verlag, 2006. It can be obtained here.
To use Palabos on the CHPC you must first install the libraries to your user directory. To do so, go to the Palabos git repository and download the latest version as a tar file. Transfer this file to your user directory and extract the tar file. You can rename the install folder (this is optional)
Assuming you have downloaded the file to your user directory, and you have logged onto the Lengau cluster
cd /mnt/lustre/users/username/ tar -xzvf palabos-master.tar.gz //This is the optional step of renaming the palabos folder// mv palabos-master palabos
The next step is to load the cmake module
module load chpc/compmech/cmake/3.15.0
To demonstrate how to compile and run a Palabos simulation, I will provide the scripts and instructions for the aneurysm example included in the palabos library. First change to the example build directory (assuming the Palabos install was renamed to palabos):
The next step is to compile the code. This can be run as a job with the script given below, or in an interactive PBS session. Note that the OpenFOAM environment is invoked as it contains all the dependencies for Palabos apart from cmake. The cmake module is added in the job script given below. If you are confident that the build will work first time, you can use this script, otherwise do it in an interactive session, following the same steps:
#!/bin/bash #PBS -P projectcode #PBS -l select=1:ncpus=24:mpiprocs=24:nodetype=haswell_reg #PBS -q smp #PBS -l walltime=01:00:00 #PBS -o /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build/stdout #PBS -e /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build/stderr #PBS -m abe #PBS -M firstname.lastname@example.org ### Source the openFOAM environment (palabos needs this for the dependencies): . /apps/chpc/compmech/CFD/OpenFOAM/OF2112 ### load the cmake module module add chpc/compmech/cmake/3.15.0 ##### Running commands # Set this environment variable explicitly. export PBS_JOBDIR=/mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build # Explicitly change to the job directory cd $PBS_JOBDIR cmake .. > cmake.out make > make.out
To run the example you must change back one directory level to this directory:
If you have not run this example before, first uncompress the geometry file:
tar xzf aneurysm.tgz
If your OpenFOAM environment is still valid, you can run your aneurysm example on a single node with the following instruction:
mpirun -np 24 ./aneurysm param.xml
To run the code as a PBS batch job over several nodes, you can use this script:
#!/bin/bash #PBS -P projectcode #PBS -l select=5:ncpus=24:mpiprocs=24 #PBS -q normal #PBS -l walltime=4:00:00 #PBS -o /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/stdout #PBS -e /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/stderr #PBS -m abe #PBS -M email@example.com ### Source the openFOAM environment (palabos needs this for the dependencies): . /apps/chpc/compmech/CFD/OpenFOAM/OF2112 ### load the cmake module module add chpc/compmech/cmake/3.15.0 ##### Running commands # Set this environment variable explicitly. export PBS_JOBDIR=/mnt/lustre/users/username/palabos/examples/showCases/aneurysm/ # Explicitly change to the job directory cd $PBS_JOBDIR nproc=`cat $PBS_NODEFILE | wc -l` ###Run the code using MPI exe=aneurysm mpirun -iface ib0 -np $nproc $exe param.xml > run.out
Palabos does not come with built-in post-processing tools. However, it makes use of some ImageMagick utilities and can also produce output files in VTK format. On Lengau there are two ImageMagick modules available:
Load one of these modules if you need ImageMagick.