3.2. Jobs¶

3.2.1. Preparing a job¶

The first step to run a job using PLAMS is to create a Job instance. You need to pick a concrete class that defines a type of job you want to run (AMSJob will be used as an example in our case) and create its instance:

myjob = AMSJob(name='myfirstjob')

Various keyword arguments (arguments of the form arg=value, like name in the example above) can be passed to a job constructor, depending on the type of your job. The following keyword arguments are common for all types of jobs:

• name – a string containing the name of the job. If not supplied, default name plamsjob is used. Job name cannot contain path separator (\ in Linux, / in Windows).
• settings – a Settings instance to be used by this job. It gets copied (using copy()) so you can pass the same instance to several different jobs and changes made afterwards won’t interfere. Any instance of Job can be also passed as a value of this argument. In that case Settings associated with the passed job are copied.
• depend – a list of jobs that need to be finished before this job can start. This is useful when you want to execute your jobs in parallel. Usually there is no need to use this argument, since dependencies between jobs are resolved automatically (see Synchronization of parallel job executions). However, sometimes one needs to explicitly specify such a dependency and depend option is meant for that.

Those values do not need to be passed to the constructor, they can be set or changed later (but they should be fixed before the job starts to run):

myjob = AMSJob()
myjob.name = 'myfirstjob'
myjob.settings.runscript.pre = 'echo HelloWorld'

Single jobs can be supplied with another keyword argument, molecule. It is supposed to be a Molecule object. Multijobs, in turn, accept a keyword argument children that stores the collection of children jobs (usually a list or a dictionary).

The most meaningful part of each job object is its Settings instance. It is used to store information about contents of the input file and the runscript as well as other tweaks of job’s behavior. Thanks to the tree-like structure of Settings this information is organized in a convenient way: the top level (myjob.settings) stores general settings, myjob.settings.input is a branch for specifying input settings, myjob.settings.runscript holds information for runscript creation and so on. Some types of jobs will make use of their own myjob.settings branches and not every kind of job will require input or runscript branches (like multijob for example). The nice thing is that all the unnecessary data present in job settings is simply ignored, so accidentally plugging settings with too much data will not cause any problem (except some cases where the whole content of some branch is used, like for example the input branch in AMSJob).

myjob.default_settings.append(sometemplate)
myjob.default_settings += [temp1, temp2]

s = Settings()
s.input.ams.Task = 'SinglePoint'
s.input.ams.UseSymmetry = 'True'
t = Settings()
t.input.ams.Task = 'GeometryOptimization'
myjob = AMSJob(...)
myjob.default_settings += [s,t]
myjob.run()

3.2.2. Running a job¶

After creating a Job instance and adjusting its settings you can finally run it. It is done by invoking the run() method, which returns a Results object:

myresults = myjob.run()

(the Results object can be also accessed as myjob.results). Various keyword arguments can be passed to run(). With jobrunner and jobmanager you can specify which JobRunner and JobManager to use for your job. If those arguments are omitted, the default instances stored in config.default_jobrunner and config.default_jobmanager are taken. All other keyword arguments are collected and stored in myjob.settings.run branch of job settings, as one flat level. They can be used later by various objects involved in running your job, for example GridRunner uses them to build the command executed to submit the runscript to the queueing system.

The following steps are taken after the run() method is called:

1. myjob.settings.run is soft-updated with run() keyword arguments (so a run() argument will never override anything already present in myjob.settings.run).
2. If a parallel JobRunner was used, a new thread is spawned and all further steps of this list happen in that thread.
3. Explicit dependencies from myjob.depend are resolved. This means waiting for all jobs listed in depend to finish.
4. Job name gets registered in the job manager and the job folder is created. If a job with the same name has been registered before, a new unique name is created.
5. Job’s prerun() method is called.
6. myjob.settings are updated with the contents of myjob.default_settings (see Default settings).
7. The hash of a job is calculated and checked (see Rerun prevention). If the same job was found as previously run, its results are copied (or linked) to the current job’s folder and run() method finishes.
8. Now the real job execution happens. If the job is a single job, an input file and a runscript are created and passed to job runner’s method call(). If the job is a multijob, run() method is called for all children jobs.
9. After the execution is finished, result files produced by the job are collected and check() is used to test if the execution was successful.
10. The job folder is cleaned using myjob.settings.keep. See Cleaning job folder for details.
11. Job’s postrun() method is called.
12. If myjob.settings.pickle is set to True, the whole job instance gets pickled and saved to the [jobname].dill file in the job folder. See Pickling for details

config.preview = True

3.2.3. Job API¶

class Job(name='plamsjob', settings=None, depend=None)[source]¶

General abstract class for all kind of computational tasks.

Methods common for all kinds of jobs are gathered here. Instances of Job should never be created. It should not be subclassed either. If you wish to define a new type of job please subclass either SingleJob or MultiJob.

Methods that are meant to be explicitly called by the user are run() and occasionally pickle(). In most cases Pickling is done automatically, but if for some reason you wish to do it manually, you can use pickle() method.

Methods that can be safely overridden in subclasses are:

• check()
• hash() (see Rerun prevention)
• prerun() and postrun() (see Prerun and postrun methods)

All other methods should remain unchanged.

Class attribute _result_type defines the type of results associated with this job. It should point to a class and it must be a Results subclass.

Every job instance has the following attributes:

Attributes adjusted automatically that should not be changed by the user:

• status – current status of the job. Possible values: created, started, registered, running, finished, crashed, failed, successful, copied, preview.
• results – a reference to a results instance. An empty instance of the type stored in _result_type is created when the job object is created.
• path – an absolute path to the job folder.
• jobmanager – a job manager associated with this job.
• parent – a pointer to the parent job if this job is a child job of some MultiJob. None otherwise.

Attributes that can be modified, but only before run() is called:

• name – the name of the job.
• settings – settings of the job.
• default_settings – see Default settings.
• depend – a list of explicit dependencies.
• _dont_pickle – additional list of this instance’s attributes that will be removed before pickling. See Pickling for details.

run(jobrunner=None, jobmanager=None, **kwargs)[source]¶

Run the job using jobmanager and jobrunner (or defaults, if None). Other keyword arguments (**kwargs) are stored in run branch of job’s settings. Returned value is the Results instance associated with this job.

Warning

This method should not be overridden.

Technical

This method does not do too much by itself. After some initial preparation it passes control to the job runner, which decides if a new thread should be started for this job. The role of the job runner is to execute three methods that make the full job life cycle: _prepare(), _execute() and _finalize(). During _execute() the job runner is called once again to execute the runscript (only in case of SingleJob).

pickle(filename=None)[source]¶

Pickle this instance and save to a file indicated by filename. If None, save to [jobname].dill in the job folder.

ok(strict=True)[source]¶

Check if the execution of this instance was successful. If needed, wait for the job to finish and then check if the status is successful (or copied).

If this method is called before job’s run() method, a warning is logged and the returned value is False. The most likely cause of such a behavior is simply forgetting about run() call. However, in complicated workflows executed in parallel, it can sometimes naturally happen that one thread is ahead of others and calls ok() before some other thread has a chance to call run(). If you’re experiencing that kind of problems, please consider using strict=False to skip the run() check. But keep in mind that skipping that check will deadlock the current thread if run() never gets called.

check()[source]¶

Check if the execution of this instance was successful. Abstract method meant for internal use.

hash()[source]¶

Calculate the hash of this instance. Abstract method meant for internal use.

prerun()[source]¶

Actions to take before the actual job execution.

This method is initially empty, it can be defined in subclasses or directly added to either the whole class or a single instance using Binding decorators.

postrun()[source]¶

Actions to take just after the actual job execution.

This method is initially empty, it can be defined in subclasses or directly added to either the whole class or a single instance using Binding decorators.

_prepare(jobmanager)[source]¶

Prepare the job for execution. This method collects steps 1-7 from Running a job. Should not be overridden. Returned value indicates if job execution should continue (Rerun prevention did not find this job as previously run).

_get_ready()[source]¶

Get ready for _execute(). This is the last step before _execute() is called. Abstract method.

_execute(jobrunner)[source]¶

Execute the job. Abstract method.

_finalize()[source]¶

Gather the results of the job execution and organize them. This method collects steps 9-12 from Running a job. Should not be overridden.

__getstate__()[source]¶

Prepare this job instance for pickling.

Attributes jobmanager, parent, default_settings and _lock are removed, as well as all attributes listed in self._dont_pickle.

_log_status(level)[source]¶

Log the status of this instance on a chosen log level. The message is uppercased to clearly stand out among other log entries.

3.2.4. Single jobs¶

class SingleJob(molecule=None, name='plamsjob', settings=None, depend=None)[source]¶

Abstract class representing a job consisting of a single execution of some external binary (or arbitrary shell script in general).

In addition to constructor arguments and attributes defined by Job, the constructor of this class accepts the keyword argument molecule that should be a Molecule instance. The constructor creates a copy of the supplied Molecule and stores it as the molecule attribute.

Class attribute _filenames defines default names for input, output, runscript and error files. If you wish to override this attribute it should be a dictionary with string keys 'inp', 'out', 'run', 'err'. The value for each key should be a string describing corresponding file’s name. Shortcut $JN can be used for job’s name. The default value is defined in the following way:

_filenames = {'inp':'$JN.in', 'run':'$JN.run', 'out':'$JN.out', 'err': '$JN.err'}

This class defines no new methods that could be directly called in your script. Methods that can and should be overridden are get_input() and get_runscript().

get_input()[source]¶

Generate the input file. Abstract method.

This method should return a single string with the full content of the input file. It should process information stored in the input branch of job settings and in the molecule attribute.

get_runscript()[source]¶

Generate the runscript. Abstract method.

This method should return a single string with the runscript contents. It can process information stored in runscript branch of job settings. In general the full runscript has the following form:

[first line defined by job.settings.runscript.shebang]

[contents of job.settings.runscript.pre, when present]

[value returned by get_runscript()]

[contents of job.settings.runscript.post, when present]

When overridden, this method should pay attention to .runscript.stdout_redirect key in job’s settings.

hash_input()[source]¶

Calculate SHA256 hash of the input file.

hash_runscript()[source]¶

Calculate SHA256 hash of the runscript.

hash()[source]¶

Calculate unique hash of this instance.

The behavior of this method is adjusted by the value of hashing key in JobManager settings. If no JobManager is yet associated with this job, default setting from config.jobmanager.hashing is used.

Methods hash_input() and hash_runscript() are used to obtain hashes of, respectively, input and runscript.

Currently supported values for hashing are:

• False or None – returns None and disables Rerun prevention.
• input – returns the hash of the input file.
• runscript – returns the hash of the runscript.
• input+runscript – returns SHA256 hash of the concatenation of hashes of input and runscript.

check()[source]¶

Check if the calculation was successful.

This method can be overridden in concrete subclasses of SingleJob. It should return a boolean value. The definition here serves as a default, to prevent crashing if a subclass does not define its own check(). It always returns True.

Warning

This method is meant for internal usage and should not be explicitly called in your script (but it can be overridden in subclasses). Manually calling check() is not thread safe. For a thread safe function to evaluate the state of your job please use ok()

full_runscript()[source]¶

Generate the full runscript, including shebang line and contents of pre and post, if any. In practice this method is just a simple wrapper around get_runscript().

_get_ready()[source]¶

Create input and runscript files in the job folder. Methods get_input() and full_runscript() are used for that purpose. Filenames correspond to entries in the _filenames attribute

_execute(jobrunner)[source]¶

Execute previously created runscript using jobrunner.

The method call() of jobrunner is used. Working directory is self.path. self.settings.run is passed as runflags argument.

If preview mode is on, this method does nothing.

classmethod load_external(path, settings=None, molecule=None, finalize=False)[source]¶

Load an external job from path.

In this context an “external job” is an execution of some external binary that was not managed by PLAMS, and hence does not have a .dill file. It can also be used in situations where the execution was started with PLAMS, but the Python process was terminated before the execution finished, resulting in steps 9-12 of Running a job not happening.

All the files produced by your computation should be placed in one folder and path should be the path to this folder. The name of the folder is used as a job name. Input, output, error and runscript files, if present, should have names defined in _filenames class attribute (usually [jobname].in, [jobname].out, [jobname].err and [jobname].run). It is not required to supply all these files, but in most cases one would like to use at least the output file, in order to use methods like grep_output() or get_output_chunk().

This method is a class method, so it is called via class object and it returns an instance of that class:

>>> j = SingleJob.load_external(path='some/path/jobname')
>>> type(j)
scm.plams.core.basejob.SingleJob
>>> a = AMSJob.load_external(path='some/path/jobname')
>>> type(a)
scm.plams.interfaces.adfsuite.ams.AMSJob

You can supply Settings and Molecule instances as settings and molecule parameters, they will end up attached to the returned job instance. If you don’t do this, PLAMS will try to recreate them automatically using methods recreate_settings() and recreate_molecule() of the corresponding Results subclass. If no Settings instance is obtained in either way, the defaults from config.job are copied.

You can set the finalize parameter to True if you wish to run the whole _finalize() on the newly created job. In that case PLAMS will perform the usual check() to determine the job status (successful or failed), followed by cleaning of the job folder (Cleaning job folder), postrun() and pickling (Pickling). If finalize is False, the status of the returned job is copied.

class MyJob(SingleJob):
    def get_input(self):
        ...
        return 'string with input file'
    def get_runscript(self):
        ...
        return 'string with runscript'

class MyJob(SingleJob):
    def __init__(self, myarg1, myarg2=default2, **kwargs):
        SingleJob.__init__(self, **kwargs)
        # do stuff with myarg1 and myarg2

3.2.5. Multijobs¶

class MultiJob(children=None, name='plamsjob', settings=None, depend=None)[source]¶

Concrete class representing a job that is a container for other jobs.

In addition to constructor arguments and attributes defined by Job, the constructor of this class accepts two keyword arguments:

• children – iterable container with children jobs (usually a list or a dictionary).
• childrunner – by default all the children jobs are run using the same JobRunner as the parent job. If you wish to use a different JobRunner for children, you can pass it using childrunner.

Values passed as children and childrunner are stored as instance attributes and can be adjusted later, but before the run() method is called.

This class defines no new methods that could be directly called in your script.

When executed, a multijob runs all its children using the same JobManager and JobRunner (unless childrunner is set). If you need to specify different run flags for children you can do it by manually setting them in child job settings:

childjob = myjob.children[0]
childjob.settings.run.arg = 'value'

Since the run branch of settings gets soft-updated with run flags, the value set this way is not overwritten by parent job.

The job folder of a multijob gets cleaned independently of its children. See Cleaning job folder for details.

Private attributes _active_children and _lock are essential for proper parallel execution. Please do not modify them.

new_children()[source]¶

Generate new children jobs.

This method is useful when some of children jobs are not known beforehand and need to be generated based on other children jobs, like for example in any kind of self-consistent procedure.

The goal of this method is to produce a new portion of children jobs. Newly created jobs should be returned in a container compatible with self.children (e.g. list for list, dict for dict). No adjustment of newly created jobs’ parent attribute is needed. This method cannot modify _active_children attribute.

The method defined here is a default template, returning None, which means no new children jobs are generated and the entire execution of the parent job consists only of running jobs initially found in self.children. To modify this behavior you can override this method in a MultiJob subclass or you can use one of Binding decorators, just like with Prerun and postrun methods.

hash()[source]¶

Hashing for multijobs is disabled by default. Returns None.

check()[source]¶

Check if the execution of this instance was successful, by calling Job.ok() of all the children jobs.

other_jobs()[source]¶

Iterate through other jobs that belong to this MultiJob, but are not in children.

Sometimes prerun() or postrun() methods create and run some small jobs that don’t end up in children collection, but are still considered a part of a MultiJob instance (their parent atribute points to the MultiJob and their working folder is inside MultiJob’s working folder). This method provides an iterator that goes through all such jobs.

Each attribute of this MultiJob that is of type Job and has it’s parent pointing to this MultiJob is returned, in a random order.

_get_ready()[source]¶

Get ready for _execute(). Count children jobs and set their parent attribute.

__iter__()[source]¶

Iterate through children. If it is a dictionary, iterate through its values.

_notify()[source]¶

Notify this job that one of its children has finished.

Decrement _active_children by one. Use _lock to ensure thread safety.

_execute(jobrunner)[source]¶

Run all children from children. Then use new_children() and run all jobs produced by it. Repeat this procedure until new_children() returns an empty list. Wait for all started jobs to finish.

mj = MultiJob(name='somejobs', children=[job1, job2, job3])
mj.children.append(job4)
mj.run(...)