Versionen im Vergleich

Schlüssel

  • Diese Zeile wurde hinzugefügt.
  • Diese Zeile wurde entfernt.
  • Formatierung wurde geändert.


Auszug

An an object-oriented Computational Fluid Dynamics(CFD) open source CFD toolkit

Description

OpenFOAM core is an extensible Computational Fluid Dynamics framework written in C++, i.e. it provides a sufficient an abstraction for a programmer to build their own code for an underlying mathematical model. 

Prerequisites

OpenFOAM is a free, open source software which is released under the GNU-GPL license

Modules

The following versions of OpenFOAM are installed in HLRN system

...

Info

The module name is openfoam. Other versions may be installed. Inspect the output of :            module avail openfoam

Modules

Codeblock
titleTo see our installed versions type:
module avail openfoam

Example Jobscripts

Codeblock
languagebash
titleSerial job for v5 based on cavity tutorial
collapsetrue
#!/bin/bash
#SBATCH --time 1:00:00
#SBATCH --nodes 1 
#SBATCH --tasks-per-node 96 
#SBATCH -p standard96:test
#SBATCH --job-name=test_job
#SBATCH --output=ol-%x.%j.out
#SBATCH --error=ol-%x.%j.err
 
export I_MPI_FALLBACK=0
export I_MPI_DEBUG=6
export I_MPI_FABRICS=shm:ofi
export I_MPI_OFI_PROVIDER=psm2
export I_MPI_PMI_LIBRARY=libpmi.so
 
module load gcc/9.2.0
module load openmpi/gcc.9/3.1.5
module load openfoam/gcc.9/5

# initialize OpenFOAM environment
#---------------------
source $WM_PROJECT_DIR/etc/bashrc
source ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions # provides fcts like runApplication
 
# set working directory
#---------------------
WORKDIR="$(pwd)"

# get and open example
#---------------------
cp -r $WM_PROJECT_DIR/tutorials/incompressible/icoFoam/cavity $WORKDIR/
cd cavity
 
# run script with several cases 
#------------------------------
./Allrun

# run single case
#--------------------------
#cd cavity
#runApplication blockMesh
#icoFoam > icoFoam.log 2>&1

...

Typically, OpenFOAM causes a lot of meta data operations. This default behavior jams no only your job but may slow down the shared parallel file system (=Lustre) for all other users. Also, your job is interrupted if the inode limit (number of files) of the quota system (hlrnquotashow-quota) is exceeded.

If you can not use our local 2TB-SSDs (see Special Filesystems) #SBATCH --partition={standard,large,huge}96:ssd at $LOCAL_TMPDIR please refer to our general advices to reduce Metadata Usage on WORK (=Lustre).

To adapt/optimize your OpenFOAM job specifically for I/O operations on $WORK (=Lustre) we strongly recommend the following steps:

  • Always, to avoid that each processor writes in its own file please use collated file I/O.
    This feature was released 2017 for all OpenFOAM versions.
    [ESI www.openfoam.com/releases/openfoam-v1712/parallel.php]
    [Foundation www.openfoam.org/news/parallel-io]

    Codeblock
    OptimisationSwitches
    {
        fileHandler collated; // all processors share a file
    }

    to the /.OpenFOAM/v#/controlDict file or per-case override in the $FOAM_CASE/system/controlDict file.

  • Always, set

    Codeblock
    runTimeModifiable false;

    to reduce I/O activity. Only set "true" (default), if it is strictly necessary to re-read dictionaries (controlDict, ...) each time step.

  • Possibly, do not save every time step:
    [www.openfoam.com/documentation/guides/latest/doc/guide-case-system-controldict.html]
    [www.cfd.direct/openfoam/user-guide/v6-controldict]

    Codeblock
    writeControl	timeStep;
    writeInterval	100;


  • Possibly, save only the latest n time steps (overwrite older ones), such as:

    Codeblock
    purgeWrite	1000;


  • Typically, only a subset of variables is needed frequently (post-processing). The full set of variables can be saved less frequently (e.g., restart purposes). This can be achieved with [https://wiki.bwhpc.de/e/OpenFoam]:

    Codeblock
    writeControl    clockTime;
    writeInterval   21600; // write ALL variables every 21600 seconds = 6 h
    
    functions
    {
        writeFields
        {
            type writeObjects;
            libs ("libutilityFunctionObjects.so");
    
            objects
            (
    	    T U // specified variables
            );
    
            outputControl timeStep;
            writeInterval 100; // write specified variables every 100 steps
        }
    }


  • In case your HLRN run job accidentally generated thousands of small files, please pack them (at least the small-size metadata files) into a single file afterwards:

    Codeblock
    tar -xvzf singlefile.tar.gz -C /folder/subfolder/location/

    Thanks a lot for your contribution making HLRN the compute systems a great place for all...

Compiling Own Code Over OpenFOAM

...