NEDAS.core.job_submitter module

class NEDAS.core.job_submitter.JobSubmitter(job_name: str = 'run', run_dir: str = '.', nproc: int = 1, offset: int = 0, parallel_mode: Literal['serial', 'mpi', 'openmp'] = 'serial', debug: bool = False, **kwargs)[source]

Bases: ABC

Run a command (shell script) on a specific computer. For a local computer this is as easy as a subprocess call to execute the command. But for large-scale HPC, this may involve submitting the job script to a scheduler and wait for its completion.

Parameters:

  • job_name (str, optional) – Name of the job. Defaults to ‘run’.

  • run_dir (str, optional) – Directory where the job will execute. Defaults to ‘.’.

  • nproc (int, optional) – Number of processors requested. Defaults to 1.

  • offset (int, optional) – Number of processors to offset from the start. Defaults to 0.

  • parallel_mode (ParallelMode, optional) – Parallelization strategy (e.g., ‘serial’, ‘mpi’, ‘openmp’). Defaults to ‘serial’.

  • debug (bool, optional) – Enables verbose logging for troubleshooting. Defaults to False.

  • **kwargs – Other arbitrary keyword arguments.

property nproc: int

Number of requested processors for the job

property offset: int

Number of processors to skip from the beginning for the job This allows different jobs to spawm the total available nproc in the allocation

abstract property nproc_avail: int

Number of available processors on a host machine This should be redefined in subclasses to machine specific behavior

abstract property execute_command: str

Execute command for running the job on the host machine, replacing ‘JOB_EXECUTE’ in ‘commands’

parse_commands(commands: str) → str[source]

Parse shell command to replace ‘JOB_EXECUTE’ with machine-specific strings.

abstractmethod run(commands: str) → None[source]

Top level run method for a job submitter.

Parameters:

commands (str) – shell commands to be run by the job submitter. In the commands string, ‘JOB_EXECUTE’ will be replaced by the correct execute_command, and ‘JOB_ARRAY_INDEX’ will be replaced by the scheduler’s index variable name to perform array jobs.