NEDAS.job_submitters.hpc module

class NEDAS.job_submitters.hpc.HPCJobSubmitter(project: str | None = None, queue: str | None = None, ppn: int | None = None, walltime: int = 3600, check_dt: int = 20, use_job_array: bool = False, array_size: int = 1, **kwargs)[source]

Bases: JobSubmitter

JobSubmitter Class customized for a HPC with a job scheduler.

Parameters:

  • project (str, optional) – Project account name for billing/allocation. Defaults to None.

  • queue (str, optional) – The submission queue name. Defaults to None.

  • ppn (int, optional) – Processors per node. Defaults to self.nproc.

  • walltime (int, optional) – Maximum execution time in seconds. Defaults to 3600.

  • check_dt (int, optional) – Time interval (sec) between status checks. Defaults to 20.

  • use_job_array (bool, optional) – Whether to utilize scheduler job arrays. Defaults to False.

  • array_size (int, optional) – Number of elements in the job array. Defaults to 1.

property ppn: int

Number of processors per compute node requested for the job

property nnode: int

Number of compute nodes for the job

property offset_node: int

Number of compute nodes to skip from the beginning

abstract property nnode_avail: int

Number of available compute nodes on a host machine

abstract property ppn_avail: int

Number of available processors per compute node

abstract property job_array_index_name: str

Job array index variable name for the host machine, replacing ‘JOB_ARRAY_INDEX’ in ‘commands’

parse_commands(commands: str) → str[source]

Parse shell command to replace ‘JOB_EXECUTE’ with machine-specific strings.

abstract property in_job_allocation: bool

Determines if a job allocation is already availalbe on the HPC If so, the job can be run as a sub step directly, otherwise will need to submit it to the queue.

check_resources() → None[source]

Checks if requested resource is available

run(commands)[source]

Top level run method for a job submitter.

Parameters:

commands (str) – shell commands to be run by the job submitter. In the commands string, ‘JOB_EXECUTE’ will be replaced by the correct execute_command, and ‘JOB_ARRAY_INDEX’ will be replaced by the scheduler’s index variable name to perform array jobs.

run_job_as_step(commands) → None[source]

Run ‘commands’ from within a job allocation Use nproc processors starting from the offset+1 processor of the allocation

abstractmethod submit_job_and_monitor(commands)[source]

Submit ‘commands’ as a job script to the scheduler on HPC and monitor for its completion.