The following instructions are courtesy of Dr. Muaaz Bhamjee of the University of Johannesburg.

Palabos is an open source CFD package based on the Lattice Boltzmann Method (LBM). Palabos is effectively a C++ library that can be used to develop CFD applications. More information can be obtained at the Palabos website.

Refer to the following book for an introduction to the LBM: M. C. Sukop and J. D. T. Thorne, Lattice Boltzmann Modelling: An Introduction for Geoscientists and Engineers, 2nd ed. Miami. FL: Springer-Verlag, 2006. It can be obtained here.

To use Palabos on the CHPC you must first install the libraries to your user directory. To do so, go to the Palabos git repository and download the latest version as a tar file. Transfer this file to your user directory and extract the tar file. You can rename the install folder (this is optional)

Assuming you have downloaded the file to your user directory, and you have logged onto the Lengau cluster

cd /mnt/lustre/users/username/
tar -xzvf palabos-master.tar.gz
//This is the optional step of renaming the palabos folder//
mv palabos-master palabos 

The next step is to load the cmake module

module load chpc/compmech/cmake/3.15.0

To demonstrate how to compile and run a Palabos simulation, I will provide the scripts and instructions for the aneurysm example included in the palabos library. First change to the example build directory (assuming the Palabos install was renamed to palabos):

cd /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build

The next step is to compile the code. This can be run as a job with the script given below, or in an interactive PBS session. Note that the OpenFOAM environment is invoked as it contains all the dependencies for Palabos apart from cmake. The cmake module is added in the job script given below. If you are confident that the build will work first time, you can use this script, otherwise do it in an interactive session, following the same steps:

1. Get into the right directory
2. Source the OpenFOAM environment
3. Load the cmake module
4. Run cmake ../
5. Run make

PalabosCompile.pbs

#!/bin/bash 
#PBS -P projectcode 
#PBS -l select=1:ncpus=24:mpiprocs=24:nodetype=haswell_reg 
#PBS -q smp
#PBS -l walltime=01:00:00 
#PBS -o /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build/stdout 
#PBS -e /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build/stderr 
#PBS -m abe 
#PBS -M username@email.com 
### Source the openFOAM environment (palabos needs this for the dependencies): 
. /apps/chpc/compmech/CFD/OpenFOAM/OF2112 
### load the cmake module 
module add chpc/compmech/cmake/3.15.0 
##### Running commands 
# Set this environment variable explicitly. 
export PBS_JOBDIR=/mnt/lustre/users/username/palabos/examples/showCases/aneurysm/build 
# Explicitly change to the job directory 
cd $PBS_JOBDIR 
cmake .. > cmake.out 
make > make.out 

To run the example you must change back one directory level to this directory:

 /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/ 

If you have not run this example before, first uncompress the geometry file:

 tar xzf aneurysm.tgz 

If your OpenFOAM environment is still valid, you can run your aneurysm example on a single node with the following instruction:

  mpirun -np 24 ./aneurysm param.xml  

To run the code as a PBS batch job over several nodes, you can use this script:

runPalabos.qsub

#!/bin/bash 
#PBS -P projectcode 
#PBS -l select=5:ncpus=24:mpiprocs=24
#PBS -q normal 
#PBS -l walltime=4:00:00 
#PBS -o /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/stdout 
#PBS -e /mnt/lustre/users/username/palabos/examples/showCases/aneurysm/stderr 
#PBS -m abe 
#PBS -M username@email.com 
### Source the openFOAM environment (palabos needs this for the dependencies): 
. /apps/chpc/compmech/CFD/OpenFOAM/OF2112 
### load the cmake module 
module add chpc/compmech/cmake/3.15.0 
##### Running commands 
# Set this environment variable explicitly. 
export PBS_JOBDIR=/mnt/lustre/users/username/palabos/examples/showCases/aneurysm/ 
# Explicitly change to the job directory 
cd $PBS_JOBDIR 
nproc=`cat $PBS_NODEFILE | wc -l` 
###Run the code using MPI 
exe=aneurysm 
mpirun -iface ib0 -np $nproc $exe param.xml > run.out

Palabos does not come with built-in post-processing tools. However, it makes use of some ImageMagick utilities and can also produce output files in VTK format. On Lengau there are two ImageMagick modules available:

chpc/compmech/ImageMagick/7.0.5
chpc/compmech/ImageMagick/image-magick-7.0.5-9-gcc-8.3.0-fvnxz3q

Load one of these modules if you need ImageMagick.

VTK files can be displayed with Paraview. This Wiki has a dedicated page for running Paraview on the cluster.