Versionen im Vergleich

Schlüssel

  • Diese Zeile wurde hinzugefügt.
  • Diese Zeile wurde entfernt.
  • Formatierung wurde geändert.

Problem

After linking a program with a netcdf library some users report error messages like this:

  ./nctest: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory

One my find this error message by searching the web. Hence, one is in good company. Here we present methods to use the tools available at HLRN to produce running binaries. First of all an important statement:

Neither the netcdf nor the hdf5 module have any effect on your binary during runtime. It is useless to load them before running a program.

The following example program can be used for testing only. Don't ask for a sophisticated science background. The example uses FORTRAN-90 notation.

download the example

To compile the code, the netcdf data- and function types must be included by a use statement. The compiler needs to know, where the fortran-modules are located. Instead of digging into installation details, we use the result of the script nf-config, which is part of the netcdf-suite. To make it available, load a netcdf module:

   module load gcc/8.3.0
   module load netcdf/gcc.8/4.7.3

Note, compiler version and netcdf version must fit, otherwise the fortran netcdf module cannot be included. We can check this with

   nf-config --fc

fortran .mod files from different major gfortran versions are incompatible.


Dynamic linking with the fortran netcdf library

Serial programms

Now you can compile

   gfortran  -c -I`nf-config --includedir` test.f90

Instead of digging even more into the installation details, we use the result of nf-config in the link step:

   gfortran -o nctest *.o `nf-config --flibs` -lm

To understand the details, just issue the commands nf-config --includedir and nf-config --flibs separately.

Now you may unload both, the compiler and the netcdf module as well. The binary runs anyway, since the path to the netcdf- and hdf5- libraries is stored in the binary. This information was extracted from the environment variable LD_RUN_PATH that is set when loading a netcdf module. Note, the path to the hdf5 libraries linked to the netcdf library is inherited by our binary. The effect of LD_RUN_PATH is broken, if the linker option rpath is used for other purposes. Say, your example program links another library located in /mybla. Checkout

   gfortran -o nctest -Wl,-rpath=/mybla *.o `nf-config --flibs` -lm  
   ldd nctest

The netcdf-libraries as well as the compiler specific libraries are not found any more. Here we notice, the compiler module also uses LD_RUN_PATH to set the path to the compiler specific libraries. So handle LD_RUN_PATH with care not to disturb this important feature! The correct way would be:

   gfortran -o nctest -Wl,-rpath="/mybla:$LD_RUN_PATH" *.o `nf-config --flibs` -lm  

Hint, you may use readelf to see more details on the internal binary structure. Watch out for the library sections.

MPI programms

Now we need to use the compiler wrappers to include mpi.h and link the mpi libraries automatically. Here, rpath is used internally, which breaks the automatic propagation of the information of the paths, where the netcdf library resides. This concerns also the location of compiler specific libraries like libgcc_s.so.1 or libquadmath.so.0 . From that there may arise problems, since libquadmath.so.0 coming with gcc.9 contains more symbols than the system version. (This allows usage of the netcdf library build with gcc.8 together with gcc.9 .) Hence, we have to use the rpath option to provide the information on the library path and propagate it to the linker. Load the module files:

  module load gcc/9.2.0
  module load netcdf/gcc.8/4.7.3
  module load openmpi/gcc.9/3.1.5

and check LD_RUN_PATH:

  echo $LD_RUN_PATH
  /sw/comm/openmpi/3.1.5/skl/gcc/lib:/sw/dataformats/netcdf/gcc.8.3.0/4.7.3/skl/lib:/sw/compiler/gcc/9.2.0/skl/lib64/

Compile:

  mpifort  -c -I`nf-config --includedir` test.f90
  mpifort -o nctest *.o `nf-config --flibs` -Wl,-rpath=$LD_RUN_PATH -lm

To check the true content of the binary, unload the modules first

  module unload openmpi/gcc.9/3.1.5
  module unload gcc/9.2.0
  module unload netcdf/gcc.8/4.7.3
  ldd nctest

With readelf -a more details can be explored:

  readelf -a nctest | less
....
Dynamic section at offset 0x2d58 contains 37 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libnetcdff.so.7]
 0x0000000000000001 (NEEDED)             Shared library: [libnetcdf.so.15]
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libmpi_usempif08.so.40]
 0x0000000000000001 (NEEDED)             Shared library: [libmpi_usempi_ignore_tkr.so.40]
 0x0000000000000001 (NEEDED)             Shared library: [libmpi_mpifh.so.40]
 0x0000000000000001 (NEEDED)             Shared library: [libmpi.so.40]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libquadmath.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/sw/comm/openmpi/3.1.5/skl/gcc/lib:/sw/dataformats/netcdf/gcc.8.3.0/4.7.3/skl/lib:/sw/compiler/gcc/9.2.0/skl/lib64/:/sw/tools/hwloc/1.1.13/skl/lib]
... 


Static linking with the fortran netcdf library

There is no simple way to find all required libraries, but information on the libraries can be gathered from the file libnetcdff.la made by libtool for dynamic linking.

   NETCDF_LIB=`nc-config --libdir`
   cat $NETCDF_LIB/libnetcdff.la | grep dependency_libs 

We configure the paths to the relevan libraries by hand:

   NETCDF_LIB=`nc-config --libdir`
   HDF5_LIB=/sw/dataformats/hdf5/1.8.20/skl/gcc.8.2.0.hlrn/lib
   SZIP_LIB=/sw/dataformats/szip/2.1/skl/gcc.8.2.0.hlrn/lib
   
   LIBS="$NETCDF_LIB/libnetcdff.a $NETCDF_LIB/libnetcdf.a $HDF5_LIB/libhdf5_hl.a  $HDF5_LIB/libhdf5.a $SZIP_LIB/libsz.a -lpthread -lcurl -lz -lm -ldl"

Now compile and link

  gfortran -fPIC -c -I`nf-config --includedir` -Bstatic test.f90
  gfortran -o nctest *.o $LIBS

and have fun.