...

Typically, OpenFOAM causes a lot of meta data operations. This default behavior jams no only your job but may slow down the shared parallel file system (=Lustre) for all other users. Also, your job is interrupted if the inode limit (number of files) of the quota system (show-quota) is exceeded.

If you can not use our local 2TB-SSDs (see Special Filesystems) #SBATCH --partition={standard,large,huge}96:ssd at $LOCAL_TMPDIR please refer to our general advices to reduce Metadata Usage on WORK (=Lustre).

To adapt/optimize your OpenFOAM job specifically for I/O operations on $WORK (=Lustre) we strongly recommend the following steps:

• Always, to avoid that each processor writes in its own file please use collated file I/O.
  This feature was released 2017 for all OpenFOAM versions.
  [ESI www.openfoam.com/releases/openfoam-v1712/parallel.php]
  [Foundation www.openfoam.org/news/parallel-io]

  Codeblock
  OptimisationSwitches { fileHandler collated; // all processors share a file }

  to the /.OpenFOAM/v#/controlDict file or per-case override in the $FOAM_CASE/system/controlDict file.

• Always, set

  Codeblock
  runTimeModifiable false;

  to reduce I/O activity. Only set "true" (default), if it is strictly necessary to re-read dictionaries (controlDict, ...) each time step.

• Possibly, do not save every time step:
  [www.openfoam.com/documentation/guides/latest/doc/guide-case-system-controldict.html]
  [www.cfd.direct/openfoam/user-guide/v6-controldict]

  Codeblock
  writeControl timeStep; writeInterval 100;

  
  

• Possibly, save only the latest n time steps (overwrite older ones), such as:

  Codeblock
  purgeWrite 1000;

  
  

• Typically, only a subset of variables is needed frequently (post-processing). The full set of variables can be saved less frequently (e.g., restart purposes). This can be achieved with [https://wiki.bwhpc.de/e/OpenFoam]:

  Codeblock
  writeControl clockTime; writeInterval 21600; // write ALL variables every 21600 seconds = 6 h functions { writeFields { type writeObjects; libs ("libutilityFunctionObjects.so"); objects ( T U // specified variables ); outputControl timeStep; writeInterval 100; // write specified variables every 100 steps } }

  
  

• In case your HLRN run job accidentally generated thousands of small files, please pack them (at least the small-size metadata files) into a single file afterwards:

  Codeblock
  tar -xvzf singlefile.tar.gz -C /folder/subfolder/location/

  Thanks a lot for your contribution making HLRN the compute systems a great place for all...

Compiling Own Code Over OpenFOAM

...