FIXME: refresh for current installation on HLRN-IV (work in progress by Tobias Kramer)
Allinea DDT from Allinea Software is a parallel debugger installed on HLRN system.
1 Introduction
Allinea DDT is a graphical parallel debugger. HLRN's current licences allow you to debug programs up to 1024 processes.
The number of threads is not taken into account - which means that you can debug several thousands of cores using MPI and OpenMP simultaneously.
You also have access to a trial licence of Allinea MAP (the profiling duration is limited to 30sec).
Allinea DDT can be used to debug several types of parallel codes (distributed or shared memory applications).
...
2- Preparing the environment
a- Loading the Allinea DDT Module
To use Allinea DDT, first load the Allinea DDT module to set the correct environment settings with the following command :
% module load ddt
If you are using the remote client, you may want to add this command line in an initialization script.
b- Compiling code to run with Allinea DDT
If you compile a program with –g, Allinea DDT can show the exact line of source code that causes the error, along with the variables and their values.
It may also be worth turning optimizations off with –O0 as these reorder the code in surprising ways:
% cc –g –O0 hello.c –o hello
3- Getting Allinea DDT started
Sometimes the quickest way to solve a problem is to debug it interactively, letting you test and explore theories of program behavior in real time.
Alternatively, you can use offline debugging (in non-interactive mode).
Interactive graphical debugging: starting Allinea DDT within an interactive session
Be sure to log in with an X window forwarding enabled. This could mean using the -X or -Y option to ssh. The -Y option often works better for Mac OS/X.
% ssh -X <username>@blogin1.hlrn.de
After loading the Allinea DDT module and compiling with the -g option, request and interactive session :
% qsub -IX -V -l walltime=00:30:00,mppwidth=32
or
% qsub -IX ./your_script.sub
where your_script.sub looks like the following :
#!/bin/bash
#PBS -l walltime=01:00:00
#PBS -l mppwidth=32
#PBS -V
#PBS -o output.log
#PBS -e error.log
Once the session starts you can load the Allinea DDT module if necessary and then run the debugger. To do so, replace the aprun command as follow :
% aprun -specific _mpi_arg -n 32 ./hello arg1 arg2
becomes
% ddt -noqueue -mpiargs "-specific_mpi_arg" -n 32 hello arg1 arg2
The Allinea DDT GUI will appear with a dialog that shows your program name and arguments, allowing you to make changes if you wish. Once ready, click “Run”.
Note : the -noqueue option is highly recommended. If not specified, Allinea DDT may try to request nodes from the workload scheduler - which will fail as this has already been done using the qsub command.
4- Trouble Shooting
If you are having trouble launching Allinea DDT try these steps.
1- Make sure you are requesting an interactive batch session on the system.
2- Make sure you compiled the code with -g
3- Make sure you have the most recent versions of the config.ddt configuration file. To do so :
% rm -rf $HOME/.allinea
If none of these tips help, please contact the support team. To help reduce the number of support iterations, please add to your email :
- a log file (starting Allinea DDT using
% ddt -debug -log output.log )
- information about the environment (version of Allinea DDT, compilers, MPI library, etc.).
The email to the support team is : support@allinea.comhlrn.de
5- Other mechanisms available to start a job
a- Interactive graphical debugging : starting Allinea DDT on the front-endlogin node
In this case, Allinea DDT will submit the job on the cluster for you.
Be sure to log in with an X window forwarding enabled. This could mean using the -X or -Y option to ssh. The -Y option often works better for Mac OS/X.
% ssh -X <username>@blogin1.hlrn.de
Make sure to load the Allinea DDT module and compile the code with the -g option.
As Allinea DDT requests the node allocation, therefore, you just need to start Allinea DDT directly on the front-end using the following command :
% ddt -n 32 hello arg1 arg2
The Allinea DDT GUI will appear with a dialog that shows your program name and arguments, allowing you to make changes if you wish. Once ready, click “Submit”.
You can also skip this preliminary step by adding the option "-start" :
% ddt -start -n 32 hello.exe arg1 arg2
b- Non-interactive debugging (offline mode)
This is the simplest way to track down a segmentation fault or core dump with Allinea DDT. This runs your job as usual but produces a detailed HTML report instead of a core file when the program finishes.
As always, compile your code accordingly and make sure to load Allinea DDT module.
Once you are ready, edit your submission script to run Allinea DDT using the following command :
% qsub ./your_script.sub
where your_script.sub looks like the following :
#!/bin/bash
#PBS -l walltime=01:00:00
#PBS -l mppwidth=128
#PBS -V
#PBS -o output.log
#PBS -e error.log
#aprun -specific _mpi_arg -n 32 ./hello arg1 arg2
ddt -noqueue -offline report.html -mpiargs "-specific_mpi_arg" -n 32 hello arg1 arg2
You can also add breakpoints and tracepoints by either :
- using the command line interface (see ddt --help)
- providing Allinea DDT with a session file, previously generated within the GUI.
6- Advanced features
a- Allinea DDT options
All the options of Allinea DDT can be found typing the command :
% ddt --help
The options you are most likely to use are :
-noqueue : ask Allinea DDT not to submit a job through the workload scheduler. This implies that you do the job submission manually (using qsub for instance).
-ddtsession <session.ddt> : provides Allinea DDT with a previously saved session file
-start : skip the pereliminary steps and start Allinea DDT with your job straight away.
-mpiargs : if you normally pass extra options to aprun other than the number of processes, you can pass these using Allinea DDT’s –mpiargs argument.:
Note that the –n 4 must be passed to the ddt command and not put inside the mpiargs.
b- Memory Debugging
Multiple mechanisms are available within Allinea DDT to monitor memory bugs. On Cray systems, it is necessary to manually link applications to a library called "dmalloc".
Typically, the command that you can use to link your application will be :
% ftn -g -dynamic -Wl,--allow-multiple-definition -L/opt/cray/ddt/4.0.1.0_32296/lib/64 -ldmallothcxx -o exe obj1.o obj2.o
Next, when DDT starts, you must click the "Memory Debugging" checkbox in the Allinea DDT run menu that first comes up.
To set detailed memory debugging options, click the 'Details...' button on the far right side, which will open the 'Memory Debugging Options' window. There you can set :
- the heap debugging level
- the number of guard pages before or after arrays for detection of heap overflow or underflow in the program, etc. The default page size is 4 KB.
Several features are enabled with memory debugging. To see them, select "Current Memory Usage" and "Memory Statistics" under the "View" menu.
c- Remote client
The Allinea DDT/MAP remote client allows you to debug and profile jobs on a remote server, while running the GUI on your local machine.
Any regular Linux build of the Allinea tools can be used as a server or as a remote client. There are also client-only builds available for OS X and Windows.
Step 1 : Download
The first step is to download a copy of the Allinea tools for your laptop or desktop. Versions for Windows, Mac OS/X and Linux are all available for download from the Allinea websitAllinea website(http://www.allinea.com/products/downloads).
Step 2 : Installation
Depending on the operating system you’re using, the instructions here differ slightly:
Linux : Extract the tarball, run installer for a GUI installer, or textinstall.sh for a text install.
Mac : Open the DMG file with finder. From here you can launch the installer.
Windows : Execute the .exe installer downloaded from the website.
Note: No licence is required for the remote client. When using the remote client, the license on the remote machine / cluster will be used.
Step 3 : Connect using the Remote Client
After installing the remote client as in step 2, run it by e.g. double-clicking on the icon.
On the welcome screen is a “Remote Launch: Off” selection box. Click on this and choose the “Configure” option.
Enter the details of your remote host:
Username and hostname: <username>@blogin1.hlrn.de
The full path of the allinea/tools installation directory on the remote machine: e.g. /opt/cray/ddt/<version>
Optional: The “script” parameter is optional and can usually be left blank. You can put the path of a script that you would like to “source” (execute and load the environment variables from) on the remote machine. Normal login scripts such as .bashrc are sourced automatically and do not need to be specified here.
If used, the path to this must be a path on the remote machine.
Click Ok
The Allinea tools connects to the remote host using SSH and may prompt you for your password or, on some systems, a PASSCODE. Log in here as you normally would, e.g. if you use an OTP token when connecting with SSH then you should use that here, too.
Once connected there may be a slight delay and then the run, attach and other options will become enabled. The interface now behaves as if you were running it directly on the remote system – all the file dialogs browse the remote filesystem, the settings are loaded and saved to the remote system and programs will be run and debugged or profiled on the remote system.
The Allinea tools remember the settings you chose, and next time you will be able to pick username@blogin1.hlrn.de from the drop-down box on the welcome dialog directly – there’s no need to type everything in again.
You will still need to SSH to the cluster to compile your program and edit files.
...
Arm DDT ("DDT") is a parallel debugger and part of the Arm Forge software suite. You can find more information at Arm DDT Get Started.
DDT is installed on HLRN IV and easiest used via a connection to a locally running GUI.
- download and install the Arm Forge Client for Mac/Windows/Linux for the same minor version as installed on HLRN (currently 20.1) at https://developer.arm.com/tools-and-software/server-and-hpc/downloads/arm-forge
- open the installed program and adapt the settings for the HLRN installation:
Click on "Configure...", next click on "Add", then add a connection name: HLRN-IV Berlin or Göttingen, put the hostname of a login node which you can reach via your ssh setup, set the remote installation directory to/sw/tools/allinea/forge-20.1.3 - adapt your job script to initiate a debugging session. We recommend to use less than 8 nodes and if possible the testing queues. Also recompile your code with debugging information enabled "
-g" and disable optimizations "-O0" to avoid reordering of instructions.
Load the modulemodule load forge/20.1.3
in you job script and prefix your srun/mpirun call withddt –connect srun app - relaunch your local client, you should receive a "Reverse connection request", which you accept. This starts your debugging session.