Starting from quarter 04/2022, the two NHR centers

NHR@ZIB and
NHR@Göttingen

follow NHR-wide regulations and account the resource usage in the new unit core hours.

Both NHR centers provide computational resources primarily to universities in Germany under public law.

A user accesses an account containing a number of units core hour.
The NHR centers charge for batch jobs on the compute systems. The charge for a job is a number of units core hour.
Usage of persistent storage including the tape library are currently not accounted.

Charge Rates

The NHR centers operate different types of compute nodes (HLRN-IV system) organized in partitions, see Hardware. The charge rates for the partitions are given in the table.

one node in partition	charged "core hours" per 1h occupancy time	increased charge rate due to
standard96 standard96:test	96
large96 large96:test large96:shared	144	high memory layout
huge96	192	high memory layout
medium40 medium40:test	40
large40 large40:test	80	high memory layout
gpu40	600	four NVidia A100 (40 GB) GPUs per node

Job Charge

The charge for a batch job on the NHR systems is the number of core hours and is calculated from the number of nodes reserved for the job, the wallclock time used by the job, and the charge rate for the job nodes. For a batch job with

num nodes,
running with a wallclock time of t hours, and
on a partition with a charge rate charge_p

the job charge charge_j yields

charge_j = num * t * charge_p

A job on 10 nodes running for 3 hours on partition huge96 (= 192 core hour) yields a job charge of 5760 core hour.

Batch jobs running in the partition large96:shared access a subset of cores on a node. For a reservation of cores, the number of nodes is the appropriate node fraction.

A job on 48 cores on partition large96:shared (96 cores per node, 144 core hour) has a reservation for

num = 48/96 = 0.5 nodes. Assuming a wallclock time of 3 hours yields a job charge of 216 core hour.

Account Types

Running batch jobs with a user account, the user can access different accounts in general. The NHR centers distinguish between

personal accounts and
project accounts

Personal Account

On the HLRN-IV systems, each user account is attached to a bank account containing core hours. Both accounts, the user account and the bank account, share the same name but their purpose is different. The user account is the account of the Linux operation system and is used for authorization and access control incl. permissions to data files. The bank account is to limit the computing time.

At the beginning of each quarter, the bank account is granted by 75.000 core hours during the lifetime of the user account. In reasonable and exceptional cases, the grant of the bank account can be extended to 300 k core hours per quarter. For that, please contact your consultant. At the end of each quarter all remaining core hours in the bank account are dropped.

Project account

A compute project holds a bank account for the project. This project account contains a compute capacity in core hours. At the beginning of each quarter the account is granted by the number of core hours following the funding decision for the given compute project. A project account holds at least 4x300 k core hours per year. Unused core hours are transferred to the subsequent quarter, but only one time.

In case of problems with your compute capacity in core hours in your project account please look at /wiki/spaces/PUB/pages/425995 or contact your consultant. This might affect the

application for additional core hours,
movement of core hours between quarters.

Select the Account in Your Batch Job

Batch jobs are submitted by a user account to the compute system. For each job the user chooses the account that will be charged by the job.

For the user account the default account for compute time is defined on the link Informationen zu Ihrer Kennung/Account information of the service portal. At the beginning of the lifetime of the user account the default account is the personal account.

The user controls the account for a job using the Slurm option --account at submit time.

To charge the account myaccount
add the following line to the job script. 
#SBATCH --account=myaccount

After job script submission the batch system checks the account for account coverage and authorizes the job for scheduling. Otherwise the job rejected, please notice the error message:

You can check the account of a job that is out of core hour.
> squeue
... myaccount ... AccountOutOfNPL ...