...

Running OpenFOAM

The following steps are followed before running OpenFOAM in parallel in a distributed processors system:

1. Adjust the settings

...

Example Jobscripts

#!/bin/bash#SBATCH --time 1:00:00
#SBATCH --nodes 1 
#SBATCH --tasks-per-node 96 
#SBATCH -p standard96
#SBATCH --job-name=of_test_job
#SBATCH --output=outlog/ol-%x.%j.out
#SBATCH --error=outlog/ol-%x.%j.err

export I_MPI_FALLBACK=0
export I_MPI_DEBUG=6
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=psm2
export I_MPI_PMI_LIBRARY=libpmi.so

#--------------------------------------

module load gcc/9.2.0
module load openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/5
# Working Directories
#---------------------
WORKDIR=$TMPDIR/openfoam# initialize OpenFOAM environment
source $WM_PROJECT_DIR/etc/bashrc
. ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions  # Tutorial run functions
#---------------

cd $WORKDIR

# run OpenFOAM in parallel
#--------------------------
runApplication decomposePar
runParallel pimpleFoam

#clean the work directory
#----------------------------
rm -rf processor*

1

Important advices running OpenFOAM on a supercomputer

OpenFOAM produces lots of small files. This default behavior jams no only your job but slows down the parallel file system (Lustre) for all HLRN users. Also, you can quickly generate more files than allowed by our quota system (hlrnquota).

To reduce this meta data strain and optimize your OpenFOAM job for this supercomputer we strongly recommend the following steps:

• Always, to avoid that each processor writes in its own file please use collated file I/O.
  This feature was released 2017 for all OpenFOAM versions.
  [ESI www.openfoam.com/releases/openfoam-v1712/parallel.php]
  [Foundation www.openfoam.org/news/parallel-io]

  Codeblock
  OptimisationSwitches { fileHandler collated; // all processors share a file }

  to the $WM_PROJECT_DIR/etc/controlDict file, or the per-user override in the /.OpenFOAM/v#/controlDict file or per-case override in the $FOAM_CASE/system/controlDict file.

• Always, set

  Codeblock
  runTimeModifiable false;

  to reduce I/O activity. Only set "true" (default), if it is strictly necessary to re-read dictionaries (controlDict, ...) each time step.

• Possibly, do not save every time step:
  [www.openfoam.com/documentation/guides/latest/doc/guide-case-system-controldict.html]
  [www.cfd.direct/openfoam/user-guide/v6-controldict]

  Codeblock
  writeControl timeStep; writeInterval 100;

  
  

• Possibly, save only the latest n time steps (overwrite older ones), such as:

  Codeblock
  purgeWrite 1000;

  
  

• Typically, only a subset of variables is needed frequently (post-processing). The full set of variables can be saved less frequently (e.g., restart purposes). This can be achieved with [https://wiki.bwhpc.de/e/OpenFoam]:

  Codeblock
  functions { writeFields { type writeObjects; libs ("libutilityFunctionObjects.so"); objects ( T U // specified variables ); outputControl timeStep; writeInterval 100; // write specified variables every 100 steps } }

  Thanks a lot for your contribution making HLRN a great place for all...

Compiling Own Code Over OpenFOAM

...