Job submitters

NEDAS can be run on various computers, from laptops to high-performance computer clusters.

NEDAS.job_submitters.get_job_submitter module

Factory function to get the JobSubmitter instance.

Based on the input args host or scheduler, if an implemented JobSubmitter subclass can be located, return an instance of that subclass. If not, will use the base JobSubmitter class instance.

param host:: Host machine name to use the machine-specific job submitter.
type host:: str, optional
param scheduler:: Scheduler type (slurm or oar).
type scheduler:: str, optional
returns:: An instance of the corresponding JobSubmitter subclass.
rtype:: JobSubmitter

NEDAS.job_submitters.base module

NEDAS.job_submitters.oar module

class NEDAS.job_submitters.oar.OARJobSubmitter(**kwargs)[source]

Bases: HPCJobSubmitter

JobSubmitter Class customized for OAR schedulers

property nproc_avail: Number of available processors on a host machine This should be redefined in subclasses to machine specific behavior

property node_list_avail

property nnode_avail: Number of available compute nodes on a host machine

property ppn_avail: Number of available processors per compute node

update_node_file()[source]

property execute_command: Execute command for running the job on the host machine, replacing ‘JOB_EXECUTE’ in ‘commands’

property job_array_index_name: Job array index variable name for the host machine, replacing ‘JOB_ARRAY_INDEX’ in ‘commands’

property in_job_allocation: bool: Determines if a job allocation is already availalbe on the HPC If so, the job can be run as a sub step directly, otherwise will need to submit it to the queue.

run_job_as_step(commands)[source]: Run ‘commands’ from within a job allocation Use nproc processors starting from the offset+1 processor of the allocation

submit_job_and_monitor(commands)[source]: Submit ‘commands’ as a job script to the scheduler on HPC and monitor for its completion.

job_submit_cmd()[source]

get_job_id(stdout)[source]

check_job_status()[source]

NEDAS.job_submitters.slurm module

class NEDAS.job_submitters.slurm.SLURMJobSubmitter(**kwargs)[source]

Bases: HPCJobSubmitter

JobSubmitter Class customized for SLURM schedulers

MAX_NTASKS = 1000000

MAX_NNODES = 1000

MAX_PPN = 1000

property nproc_avail: Number of available processors on a host machine This should be redefined in subclasses to machine specific behavior

property nnode_avail: Number of available compute nodes on a host machine

property ppn_avail: Number of available processors per compute node

property execute_command: Execute command for running the job on the host machine, replacing ‘JOB_EXECUTE’ in ‘commands’

property job_array_index_name: Job array index variable name for the host machine, replacing ‘JOB_ARRAY_INDEX’ in ‘commands’

property in_job_allocation: bool: Determines if a job allocation is already availalbe on the HPC If so, the job can be run as a sub step directly, otherwise will need to submit it to the queue.

submit_job_and_monitor(commands)[source]: Submit ‘commands’ as a job script to the scheduler on HPC and monitor for its completion.