===General Info===

LAMMPS ("Large-scale Atomic/Molecular Massively Parallel Simulator") is a molecular dynamics program from Sandia National Laboratories.  LAMMPS makes use of MPI for parallel communication and is free, open-source software, distributed under the terms of the GNU General Public License. [[https://en.wikipedia.org/wiki/LAMMPS|source]]

===Useful Links===

Developer's Benchmarks [[http://lammps.sandia.gov/bench.html]] \\
HPCAC Best Practices [[http://www.hpcadvisorycouncil.com/pdf/LAMMPS_Best_Practice.pdf]]

===Installation Guide===

Setup environment:

  mkdir LAMMPS
  cd LAMMPS
  makedir tars

  vim bashrc

add:

  source /opt/intel/bin/compilervars.sh intel64
  source /opt/intel/mkl/bin/mklvars.sh intel64
  source /opt/intel/impi/4.1.0/bin64/mpivars.sh
  export PATH=$PATH:/usr/local/cuda/bin
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/lib
  export C_INCLUDE_PATH=/usr/local/cuda/include

  source bashrc
  cd tar

==Building the benchmark==

Download the LAMMPS source (15 May 2015) here: [[http://www.ace.chpc.ac.za/tars/lammps-stable.tar.gz|lammps-stable.tar.gz]]

  tar -xf lammps.stable.tar.gz
  mv lammps-15May15 ..
  cd ../lammps-15May15/

=Building x86 CPU Benchmark=

To build the Intel compiled LAMMPS binary, first edit the Makefile:

  cd src
  cp MAKE/OPTIONS/Makefile.intel_cpu MAKE/Makefile.intel_cpu
  vim MAKE/Makefile.intel.cpu

edit:
  FFT_INC =       -DFFT_MKL

Build with:

  make yes-user-intel
  make yes-user-omp 
  make intel_cpu
  
The lmp_intel_cpu binary should be produced.

  cp lmp_intel_cpu ../bench

==Building CUDA GPU Benchmark==

Build this binary using the python script:
  cd ~/LAMMPS/lammps-15May15/bench/KEPLER

Edit the build.py file:
  vim build.py

Edit the lmp_dir variable in line 21:
  lmpdir= "/home/$USER/LAMMPS/lammps-15May15"

Correct the whitespace error in line 79:
  cpu = opt = omp = 0

Next, edit the CUDA Makefile
  vim Makefile.cuda

edit:

  CC =            mpiicpc
  CCFLAGS =       -O3 -xHost
  SHFLAGS =       -fPIC
  DEPFLAGS =      -M

  LINK =          mpiicpc
  LINKFLAGS =     -O3 -xHost
  LIB =           -lstdc++
  SIZE =          size

  LMP_INC =       -DLAMMPS_GZIP -DLAMMPS_JPEG

  MPI_INC =       -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1
  MPI_PATH =
  MPI_LIB =

  FFT_INC = 
  FFT_PATH = 
  FFT_LIB = 

Build the binary with:

  python build.py cuda
  cp lmp_cuda ../
  
===Benchmark===

The stock "3d Lennard-Jones melt" test problem is used as a benchmark for this code.
A fixed number of particles/core is used for the problem size.


cd ~/LAMMPS/lammps-15May15/bench

A pre-configured input script is available here {{http://wiki.chpc.ac.za/_media/acelab:cpu.tar.gz|cpu.tar.gz}}

==x86 CPU Benchmark==

For x86 CPU benchmarks, this is **500K particles per core.**
Therefore, if you with to run the benchmark on 24 x86 cores, a total of 12,000K (500x24) particles is required.

The run the benchmark use:
  mpirun -np <N> -hostfile <HF> ./lmp_intel_cpu -sf intel -v x <X> -v y <Y> -v z <Z> -v t 100 < in.lj

where: <N> is the number of cores, <HF> is the hostfile amd <X>, <Y> and <Z> are the problem scaling factors - used to reach 500K particles/core.
The benchmark has been pre-configured to operate using 500K particles. Therefore, the X,Y and Z values are using to scale the number of particles up - for more cores.
In order to run the benchmark on more cores simply scale X,Y and Z accordingly, such that their product equals the number of cores desired.

For example, running on 4 nodes with 24 cores each, the run command would be:
  mpirun -np 96 -hostfile hosts ./lmp_intel_cpu -sf intel -v x 6 -v y 4 -v z 4 -v t 100 < in.lj

Giving a total particle count of 500K * (6*4*4) = 48x10^6.  Which conforms to the 500K particles per core, (48x10^6 / 96 = 500K).

==CUDA GPU Benchmark==

For the GPU benchmark, the same process is used as above, however with a larger number of particles per GPU.
For GPU benchmarks, the particles per GPU are set as **8M particles per GPU.**
Download the pre-configured input file here 

  mpirun -n <N> -hostfile <HF> ./lmp_cuda_mixed -c on -sf cuda -pk cuda 1 -v x 1 -v y 1 -v z 1 < in.lj


The results of the benchmark are reported as particle-timesteps per second.  To calculate this value, take the number of particles in the simulation, multiple by simulation timesteps and divide by the runtime. See below for an example:

  500 000 [particles] * 300 [simulation timesteps] / 25.4 [seconds] = 5.9 x10^6 [particle-timesteps/second]