3.2. Jobs¶

3.2.1. Preparing a job¶

The first step to run a job using PLAMS is to create a job object. You need to pick a concrete class that defines a type of job you want to run (ADFJob will be used as an example in our case) and create its instance:

>>> myjob = ADFJob(name='myfirstjob')

Various keyword arguments (arguments of the form arg=value, like name in the example above) can be passed to a job constructor, depending on the type of your job. However, the following keyword arguments are common for all types of jobs:

• name – a string containing the name of the job. If not supplied, default name plamsjob is used. Job’s name cannot contain path separator (\ in Linux, / in Windows).
• settings – a Settings instance to be used by this job. It gets copied (using copy()) so you can pass the same instance to several different jobs and changes made afterwards won’t interfere. Any instance of Job can be also passed as a value of this argument. In that case Settings associated with the passed job are copied.
• depend – a list of jobs that need to be finished before this job can start. This is useful when you want to execute your jobs in parallel. Usually there is no need to use this argument, since dependencies between jobs are resolved automatically (see Synchronization of parallel job executions). However, sometimes one needs to explicitly state such a dependency and this option is then helpful.

Those values do not need to be passed to the constructor, they can be set or changed later (but they should be fixed before the job starts to run):

>>> myjob = ADFJob()
>>> myjob.name = 'myfirstjob'
>>> myjob.settings.runscript.pre = 'echo HelloWorld'

Single jobs can be supplied with another keyword argument, molecule. It is supposed to be a Molecule object. Multijobs, in turn, accept keyword argument children that stores the list of children jobs.

The most meaningful part of each job object is its Settings instance. It is used to store information about contents of job’s input file, runscript as well as other tweaks of job’s behavior. Thanks to tree-like structure of Settings, this information is organized in a convenient way: the top level (myjob.settings.) stores general settings, myjob.settings.input. is a branch for specifying input settings, myjob.settings.runscript. holds information for runscript creation and so on. Some types of jobs will make use of their own myjob.settings branches and not every kind of job will always require input or runscript branches (like multijob for example). The nice thing is that all the unnecessary data present in job’s settings is simply ignored, so accidentally plugging settings with too much data will not cause any problem (except some cases where the whole content of some branch is used, like for example the input branch in SCMJob).

>>> myjob.default_settings.append(sometemplate)
>>> myjob.default_settings += [temp1, temp2]

3.2.2. Running a job¶

After creating a job instance and adjusting its settings you can finally run it. It is done by invoking job’s run() method, which returns a Results instance:

>>> myresults = myjob.run()

Again, various keyword arguments can be passed here. With jobrunner and jobmanager you can specify which JobRunner and JobManager to use for your job. If those arguments are omitted, the default instances stored in config.default_jobrunner and config.jm are taken. All other keyword arguments passed here are collected and stored in myjob.settings.run branch as one flat level. They can be used later by various objects involved in running your job, for example GridRunner uses them to build command executed to submit runscript to the queueing system.

The following steps are taken after the run() method is called:

1. myjob.settings.run is soft-updated with run() keyword arguments.
2. If a parallel JobRunner was used, a new thread is spawned and all further steps of this list happen in this thread.
3. Explicit dependencies from myjob.depend are resolved. This means waiting for all jobs listed there to finish.
4. Job’s name gets registered in the job manager and the job folder is created.
5. Job’s prerun() method is called.
6. myjob.settings are updated according to contents of myjob.default_settings.
7. The hash of a job is calculated and checked (see Rerun prevention). If the same job was found as previously run, its results are copied (or linked) to the current job’s folder and run() finishes.
8. Now the real job execution happens. If your job is a single job, an input file and a runscript are produced and passed to job runner’s method call(). In case of multijob, run() method is called for all children jobs.
9. After the execution is finished, result files produced by the job are collected and check() is used to test if the execution was successful.
10. The job folder is cleaned using myjob.settings.keep. See Cleaning job folder for details.
11. Job’s postrun() method is called.
12. If myjob.settings.pickle is true, the whole job instance gets pickled and saved to the [jobname].dill file in the job folder.

>>> config.preview = True

3.2.3. Job API¶

class Job(name='plamsjob', settings=None, depend=None)[source]¶

General abstract class for all kind of computational tasks.

Methods common for all kinds of jobs are gathered here. Instances of Job should never be created. It should not be subclassed either. If you wish to define a new type of job please subclass either SingleJob or MultiJob.

Methods that are meant to be explicitly called by the user are run() and occasionally pickle(). In most cases Pickling is done automatically, but if for some reason you wish to do it manually, you can use pickle() method.

Methods that can be safely overridden in subclasses are:

• check()
• hash() (see Rerun prevention)
• prerun() and postrun() (see Prerun and postrun methods)

Other methods should remain unchanged.

Class attribute _result_type defines the type of results associated with this job. It should point to a class and it must be a Results subclass.

Every job instance has the following attributes. Values of these attributes are adjusted automatically and should not be set by the user:

• status – current status of the job in human-readable format.
• results – reference to a results instance. An empty instance of the type stored in _result_type is created when the job constructor is called.
• path – an absolute path to the job folder.
• jobmanager – a job manager associated with this job.
• parent – a pointer to the parent job if this job is a child job of some MultiJob. None otherwise.

These attributes can be modified, but only before run() is called:

• name – the name of the job.
• settings – settings of the job.
• default_settings – see Default settings.
• depend – a list of explicit dependencies.
• _dont_pickle – additional list of this instance’s attributes that will be removed before pickling. See Pickling for details.

__getstate__()[source]¶

Prepare an instance for pickling.

Attributes jobmanager, parent, default_settings and _lock are removed, as well as all attributes listed in self._dont_pickle.

run(jobrunner=None, jobmanager=None, **kwargs)[source]¶

Run the job using jobmanager and jobrunner (or defaults, if None). Other keyword arguments (**kwargs) are stored in run branch of job’s settings. Returned value is the Results instance associated with this job.

Note

This method should not be overridden.

Technical

This method does not do too much by itself. After simple initial preparation it passes control to job runner, which decides if a new thread should be started for this job. The role of the job runner is to execute three methods that make the full job life cycle: _prepare(), _execute() and _finalize(). During _execute() the job runner is called once again to execute the runscript (only in case of SingleJob).

pickle(filename=None)[source]¶

Pickle this instance and save to a file indicated by filename. If None, save to [jobname].dill in the job folder.

check()[source]¶

Check if the calculation was successful.

This method can be overridden in concrete subclasses for different types of jobs. It should return a boolean value.

The definition here serves as a default, to prevent crashing if a subclass does not define its own check(). It always returns True.

hash()[source]¶

Calculate the hash of this instance. Abstract method.

prerun()[source]¶

Actions to take before the actual job execution.

This method is initially empty, it can be defined in subclasses or directly added to either whole class or a single instance using Binding decorators.

postrun()[source]¶

Actions to take just after the actual job execution.

This method is initially empty, it can be defined in subclasses or directly added to either whole class or a single instance using Binding decorators.

_prepare(jobmanager)[source]¶

Prepare the job for execution. This method collects steps 1-7 from Running a job. Should not be overridden. Returned value indicates if job execution should continue (Rerun prevention did not find this job previously run).

_get_ready()[source]¶

Get ready for _execute(). This is the last step before _execute() is called. Abstract method.

_execute(jobrunner)[source]¶

Execute the job. Abstract method.

_finalize()[source]¶

Gather the results of job execution and organize them. This method collects steps 9-12 from Running a job. Should not be overridden.

3.2.4. Single jobs¶

class SingleJob(molecule=None, name='plamsjob', settings=None, depend=None)[source]¶

Abstract class representing a job consisting of a single execution of some external binary (or arbitrary shell script in general).

In addition to constructor arguments and attributes defined by Job, the constructor of this class accepts the keyword argument molecule that should be a Molecule instance.

Class attribute _filenames defines default names for input, output, runscript and error files. If you wish to override this attribute it should be a dictionary with string keys 'inp', 'out', 'run', 'err'. The value for each key should be a string describing corresponding file’s name. Shortcut $JN can be used for job’s name. The default value is defined in the following way:

>>> _filenames = {'inp':'$JN.in', 'run':'$JN.run', 'out':'$JN.out', 'err': '$JN.err'}

This class defines no new methods that could be directly called in your script. Methods that can and should be overridden are get_input() and get_runscript().

_filename(t)[source]¶

Return filename for file of type t. t can be any key from _filenames dictionary. $JN is replaced with job name in returned string.

get_input()[source]¶

Generate the input file. Abstract method.

This method should return a single string with full content of the input file. It should process information stored in input branch of job’s settings and in molecule attribute.

get_runscript()[source]¶

Generate runscript. Abstract method.

This method should return a single string with runscript contents. It can process information stored in runscript branch of job’s settings. In general the full runscript has the following form:

[first line defined by job.settings.runscript.shebang]

[contents of job.settings.runscript.pre, if any]

[value returned by get_runscript()]

[contents of job.settings.runscript.post, if any]

When overridden, this method should pay attention to .runscript.stdout_redirect key in job’s settings.

hash()[source]¶

Calculate unique hash of this instance.

The behavior of this method is adjusted by the value of hashing key in JobManager settings. If no JobManager is yet associated with this job, default setting from config.jobmanager.hashing is used.

Currently supported values for hashing are:

• False or None – returns None and disables Rerun prevention.
• input – returns SHA256 hash of the input file.
• runscript – returns SHA256 hash of the runscript.
• input+runscript – returns SHA256 hash of the concatenation of input and runscript.

_full_runscript()[source]¶

Generate full runscript, including shebang line and contents of pre and post, if any.

Technical

In practice this method is just a wrapper around get_runscript().

_get_ready()[source]¶

Generate input and runscript files in the job folder. Methods get_input() and get_runscript() are used for that purpose.

_execute(jobrunner)[source]¶

Execute previously created runscript using jobrunner.

The method call() of jobrunner is used. Working directory is self.path. self.settings.run is passed as runflags argument.

If preview mode is on, this method does nothing.

>>> class MyJob(SingleJob):
>>>     def get_input(self):
>>>         ...
>>>         return 'string with input file'
>>>     def get_runscript(self):
>>>         ...
>>>         return 'string with runscript'

>>> class MyJob(SomeOtherJob):
>>>     def __init__(self, myarg1, myarg2=default2, **kwargs):
>>>         SomeOtherJob.__init__(**kwargs)
>>>         # do stuff with myarg1 and myarg2

3.2.5. Multijobs¶

class MultiJob(children=None, name='plamsjob', settings=None, depend=None)[source]¶

Concrete class representing a job that is a container for other jobs.

In addition to constructor arguments and attributes defined by Job, the constructor of this class accepts two keyword arguments:

• children – should be a list (or other iterable container) containing children jobs.
• childrunner – by default all the children jobs are run using the same JobRunner as the parent job. If you wish to use a different JobRunner for children, you can pass it using this argument.

Values passed as children and childrunner are stored as instance attributes and can be adjusted later, but before the run() method is called.

This class defines no new methods that could be directly called in your script.

When executed, a multijob runs all its children using the same run() arguments. If you need to specify different run flags for children you can do it by manually setting them in children job Settings:

>>> childjob.settings.run.arg = 'value'

Since run branch of settings gets soft-updated by run flags, value set this way is not overwritten by parent job.

Job folder of a multijob gets cleaned independently of its children. See Cleaning job folder for details.

new_children()[source]¶

Generate new children jobs.

This method is useful when some of children jobs are not known beforehand and need to be generated based on other children jobs, like for example in any kind of self-consistent procedure.

The goal of this method is to produce new portion of children jobs. Newly created jobs have to be manually added to self.children and, besides that, returned as a list by this method. No adjustment of newly created jobs’ parent attribute is needed. This method cannot modify _active_children attribute.

The method defined here is a default template, returning an empty list, which means no new children jobs are generated and the entire execution of the parent job consists only of running jobs initially found in self.children. To modify this behavior you can override this method in MultiJob subclass or use one of Binding decorators, just like with Prerun and postrun methods.

hash()[source]¶

Hashing for multijobs is disabled by default. Return None.

check()[source]¶

Check if the calculation was successful. Returns True if every children job has its status attribute set to 'successful'.

_get_ready()[source]¶

Get ready for _execute(). Count children jobs and set their parent attribute.

__iter__()[source]¶

Iterate through children. If it is a dictionary, iterate through its values.

_notify()[source]¶

Notify this job that one of its children has finished.

Decrement _active_children by one. Use _lock to ensure thread safety.

_execute(jobrunner)[source]¶

Run all children from children. Then use new_children() and run all jobs produced by it. Repeat this procedure until new_children() returns an empty list. Wait for all started jobs to finish.

>>> mj = MultiJob(name='somejobs', children=[job1, job2, job3])
>>> mj.children.append(job4)
>>> mj.run(...)