Versionen im Vergleich

Schlüssel

  • Diese Zeile wurde hinzugefügt.
  • Diese Zeile wurde entfernt.
  • Formatierung wurde geändert.


Auszug

An an object-oriented Computational Fluid Dynamics(CFD) open source CFD toolkit

Description

OpenFOAM core is an extensible Computational Fluid Dynamics framework written in C++, i.e. it provides a sufficient an abstraction for a programmer to build their own code for an underlying mathematical model. 

Prerequisites

OpenFOAM is a free, open source software which is released under the GNU-GPL license

Modules

The following versions of OpenFOAM are installed in HLRN system

...

Modules

Codeblock
titleTo see our installed versions type:
module avail openfoam

Example Jobscripts

Codeblock
languagebash
titleSerial job for v5 based on cavity tutorial
collapsetrue
#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 1 
#SBATCH --tasks-per-node 96 
#SBATCH -p standard96:test
#SBATCH --job-name=test_job
#SBATCH --output=ol-%x.%j.out
#SBATCH --error=ol-%x.%j.err
 
export I_MPI_FALLBACK=0
export I_MPI_DEBUG=6
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=psm2
export I_MPI_PMI_LIBRARY=libpmi.so
 
module load gcc/9.2.0
module load openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/5

# initialize OpenFOAM environment
#---------------------
source $WM_PROJECT_DIR/etc/bashrc
source ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions # provides fcts like runApplication
 
# set working directory
#---------------------
WORKDIR="$(pwd)"

# get and open example
#---------------------
cp -r $WM_PROJECT_DIR/tutorials/incompressible/icoFoam/cavity $WORKDIR/
cd cavity
 
# run script with several cases 
#------------------------------
./Allrun

# run single case
#--------------------------
#cd cavity
#runApplication blockMesh
#icoFoam > icoFoam.log 2>&1

...

Some important advice when running OpenFOAM on a supercomputer

OpenFOAM produces lots of small filesTypically, OpenFOAM causes a lot of meta data operations. This default behavior jams no only your job but slows may slow down the shared parallel file system (=Lustre) for all HLRN other users. Also, you can quickly generate more files than allowed by our your job is interrupted if the inode limit (number of files) of the quota system (hlrnquotashow-quota) is exceeded.

If you can not use our local 2TB-SSDs (see Special Filesystems) #SBATCH --partition={standard,large,huge}96:ssd at $LOCAL_TMPDIR please refer to our general advices to reduce Metadata Usage on WORK (=Lustre).

To adapt/optimize your OpenFOAM job specifically for I/O operations on $WORK (=Lustre) we strongly recommend the following steps:

  • Always, to avoid that each processor writes in its own file please use collated file I/O.
    This feature was released 2017 for all OpenFOAM versions.
    [ESI www.openfoam.com/releases/openfoam-v1712/parallel.php]
    [Foundation www.openfoam.org/news/parallel-io]

    Codeblock
    OptimisationSwitches
    {
        fileHandler collated; // all processors share a file
    }

    to the /.OpenFOAM/v#/controlDict file or per-case override in the $FOAM_CASE/system/controlDict file.

  • Always, set

    Codeblock
    runTimeModifiable false;

    to reduce I/O activity. Only set "true" (default), if it is strictly necessary to re-read dictionaries (controlDict, ...) each time step.

  • Possibly, do not save every time step:
    [www.openfoam.com/documentation/guides/latest/doc/guide-case-system-controldict.html]
    [www.cfd.direct/openfoam/user-guide/v6-controldict]

    Codeblock
    writeControl	timeStep;
    writeInterval	100;


  • Possibly, save only the latest n time steps (overwrite older ones), such as:

    Codeblock
    purgeWrite	1000;


  • Typically, only a subset of variables is needed frequently (post-processing). The full set of variables can be saved less frequently (e.g., restart purposes). This can be achieved with [https://wiki.bwhpc.de/e/OpenFoam]:

    Codeblock
    writeControl    clockTime;
    writeInterval   21600; // write ALL variables every 21600 seconds = 6 h
    
    functions
    {
        writeFields
        {
            type writeObjects;
            libs ("libutilityFunctionObjects.so");
    
            objects
            (
    	    T U // specified variables
            );
    
            outputControl timeStep;
            writeInterval 100; // write specified variables every 100 steps
        }
    }


  • In case your HLRN run job accidentally generated thousands of small files, please pack them (at least the small-size metadata files) into a single file afterwards:

    Codeblock
    tar -xvzf singlefile.tar.gz -C /folder/subfolder/location/

    Thanks a lot for your contribution making HLRN the compute systems a great place for all...

Compiling Own Code Over OpenFOAM

...