Settings¶

Tree-like structure¶

The Settings class is based on the regular Python dictionary (built-in class dict, tutorial can be found here) and in many aspects works just like it:

>>> s = Settings()
>>> s['abc'] = 283
>>> s[147147] = 'some string'
>>> print(s['abc'])
283
>>> del s[147147]

The main difference is that data in Settings can be stored in multilevel fashion, whereas an ordinary dictionary is just a flat structure of key-value pairs. That means a sequence of keys can be used to store a value. In the example below s['a'] is itself a Settings instance with two key-value pairs inside:

>>> s = Settings()
>>> s['a']['b'] = 'AB'
>>> s['a']['c'] = 'AC'
>>> s['x']['y'] = 10
>>> s['x']['z'] = 13
>>> s['x']['foo'][123] = 'even deeper'
>>> s['x']['foo']['bar'] = 183
>>> print(s)
a:
  b:    AB
  c:    AC
x:
  foo:
      123:  even deeper
      bar:  183
  y:    10
  z:    13
>>> print(s['x'])
foo:
    123:    even deeper
    bar:    183
y:  10
z:  13

So for each key the value can be either a “proper value” (string, number, list etc.) or another Settings instance that creates a new level in the data hierarchy. That way similar information can be arranged in subgroups that can be copied, moved and updated together. It is convenient to think of a Settings object as a tree. The root of the tree is the top instance (s in the above example), “proper values” are stored in leaves (a leaf is a childless node) and internal nodes correspond to nested Settings instances (we will call them branches). Tree representation of s from the example above is illustrated on the following picture:

Tree-like structure could also be achieved with regular dictionaries, but in a rather cumbersome way:

>>> d = dict()
>>> d['a'] = dict()
>>> d['a']['b'] = dict()
>>> d['a']['b']['c'] = dict()
>>> d['a']['b']['c']['d'] = 'ABCD'
===========================
>>> s = Settings()
>>> s['a']['b']['c']['d'] = 'ABCD'

In the last line of the above example all intermediate Settings instances are created and inserted automatically. Such a behavior, however, has some downsides – every time you request a key that is not present in a particular Settings instance (for example as a result of a typo), a new empty instance is created and inserted as a value of this key. This is different from dictionaries where exception is raised in such a case:

>>> d = dict()
>>> d['foo'] = 'bar'
>>> x = d['fo']
KeyError: 'fo'
===========================
>>> s = Settings()
>>> s['foo'] = 'bar'
>>> x = s['fo']

>>> print(s)
fo:            #the value here is an empty Settings instance
foo:    bar

Dot notation¶

To avoid inconvenient punctuation, keys stored in Settings can be accessed using the dot notation in addition to the usual bracket notation. In other words s.abc works as a shortcut for s['abc']. Both notations can be used interchangeably:

>>> s.a.b = 'AB'
>>> s['a'].c = 'AC'
>>> s.x['y'] = 10
>>> s['x']['z'] = 13
>>> s['x'].foo[123] = 'even deeper'
>>> s.x.foo.bar = 183
>>> print(s)
a:
  b:    AB
  c:    AC
x:
  foo:
      123:  even deeper
      bar:  183
  y:    10
  z:    13

Due to the internal limitation of the Python syntax parser, keys other than single word strings cannot work with that shortcut, for example:

>>> s.123.b.c = 12
SyntaxError: invalid syntax
>>> s.q we.r.t.y = 'aaa'
SyntaxError: invalid syntax
>>> s.5fr = True
SyntaxError: invalid syntax

In those cases one has to use the regular bracket notation:

>>> s[123].b.c = 12
>>> s['q we'].r.t.y = 'aaa'
>>> s['5fr'] = True

The dot shortcut does not work for keys which begin and end with two (or more) underscores (like __key__). This is done on purpose to ensure that Python magic methods work properly.

Case sensitivity¶

Settings are case-preserving but case-insensitive. That means every key is stored in its original form, but when looked up (for example, to access value, test existence or delete), any casing can be used:

>>> s = Settings()
>>> s.foo = 'bar'
>>> s.System.one = 1
>>> s.system.two = 2
>>> print(s.FOO)
bar
>>> 'Foo' in s
True
>>> print(s)
foo:    bar
System:
   one:     1
   two:     2
>>> 'oNe' in s.SYSTEM
True

API¶

class Settings(*args, **kwargs)[source]¶

Automatic multi-level dictionary. Subclass of built-in class dict.

The shortcut dot notation (s.basis instead of s['basis']) can be used for keys that:

• are strings

• don’t contain whitespaces

• begin with a letter or an underscore

• don’t both begin and end with two or more underscores.

Iteration follows lexicographical order (via sorted() function)

Methods for displaying content (__str__() and __repr__()) are overridden to recursively show nested instances in easy-readable format.

Regular dictionaries (also multi-level ones) used as values (or passed to the constructor) are automatically transformed to Settings instances:

>>> s = Settings({'a': {1: 'a1', 2: 'a2'}, 'b': {1: 'b1', 2: 'b2'}})
>>> s.a[3] = {'x': {12: 'q', 34: 'w'}, 'y': 7}
>>> print(s)
a:
  1:    a1
  2:    a2
  3:
    x:
      12:   q
      34:   w
    y:  7
b:
  1:    b1
  2:    b2

__init__(*args, **kwargs)[source]¶

copy()[source]¶

Return a new instance that is a copy of this one. Nested Settings instances are copied recursively, not linked.

In practice this method works as a shallow copy: all “proper values” (leaf nodes) in the returned copy point to the same objects as the original instance (unless they are immutable, like int or tuple). However, nested Settings instances (internal nodes) are copied in a deep-copy fashion. In other words, copying a Settings instance creates a brand new “tree skeleton” and populates its leaf nodes with values taken directly from the original instance.

This behavior is illustrated by the following example:

>>> s = Settings()
>>> s.a = 'string'
>>> s.b = ['l','i','s','t']
>>> s.x.y = 12
>>> s.x.z = {'s','e','t'}
>>> c = s.copy()
>>> s.a += 'word'
>>> s.b += [3]
>>> s.x.u = 'new'
>>> s.x.y += 10
>>> s.x.z.add(1)
>>> print(c)
a:  string
b:  ['l', 'i', 's', 't', 3]
x:
  y:    12
  z:    set([1, 's', 'e', 't'])
>>> print(s)
a:  stringword
b:  ['l', 'i', 's', 't', 3]
x:
  u:    new
  y:    22
  z:    set([1, 's', 'e', 't'])

This method is also used when copy.copy() is called.

soft_update(other)[source]¶

Update this instance with data from other, but do not overwrite existing keys. Nested Settings instances are soft-updated recursively.

In the following example s and o are previously prepared Settings instances:

>>> print(s)
a:  AA
b:  BB
x:
  y1:   XY1
  y2:   XY2
>>> print(o)
a:  O_AA
c:  O_CC
x:
  y1:   O_XY1
  y3:   O_XY3
>>> s.soft_update(o)
>>> print(s)
a:  AA        #original value s.a not overwritten by o.a
b:  BB
c:  O_CC
x:
  y1:   XY1   #original value s.x.y1 not overwritten by o.x.y1
  y2:   XY2
  y3:   O_XY3

Other can also be a regular dictionary. Of course in that case only top level keys are updated.

Shortcut A += B can be used instead of A.soft_update(B).

update(other)[source]¶

Update this instance with data from other, overwriting existing keys. Nested Settings instances are updated recursively.

In the following example s and o are previously prepared Settings instances:

>>> print(s)
a:  AA
b:  BB
x:
  y1:   XY1
  y2:   XY2
>>> print(o)
a:  O_AA
c:  O_CC
x:
  y1:   O_XY1
  y3:   O_XY3
>>> s.update(o)
>>> print(s)
a:  O_AA        #original value s.a overwritten by o.a
b:  BB
c:  O_CC
x:
  y1:   O_XY1   #original value s.x.y1 overwritten by o.x.y1
  y2:   XY2
  y3:   O_XY3

Other can also be a regular dictionary. Of course in that case only top level keys are updated.

merge(other)[source]¶

Return new instance of Settings that is a copy of this instance soft-updated with other.

Shortcut A + B can be used instead of A.merge(B).

find_case(key)[source]¶

Check if this instance contains a key consisting of the same letters as key, but possibly with different case. If found, return such a key. If not, return key.

get(key, default=None)[source]¶

Like regular get, but ignore the case.

pop(key, *args)[source]¶

Like regular pop, but ignore the case.

popitem()[source]¶

Like regular popitem, but ignore the case.

setdefault(key, default=None)[source]¶

Like regular setdefault, but ignore the case and if the value is a dict, convert it to Settings.

as_dict()[source]¶

Return a copy of this instance with all Settings replaced by regular Python dictionaries.

classmethod suppress_missing()[source]¶

A context manager for temporary disabling the Settings.__missing__() magic method: all calls now raising a KeyError.

As a results, attempting to access keys absent from an arbitrary Settings instance will raise a KeyError, thus reverting to the default dictionary behaviour.

Note

The Settings.__missing__() method is (temporary) suppressed at the class level to ensure consistent invocation by the Python interpreter. See also special method lookup.

Example:

>>> s = Settings()

>>> with s.suppress_missing():
...     s.a.b.c = True
KeyError: 'a'

>>> s.a.b.c = True
>>> print(s.a.b.c)
True

get_nested(key_tuple, suppress_missing=False)[source]¶

Retrieve a nested value by, recursively, iterating through this instance using the keys in key_tuple.

The Settings.__getitem__() method is called recursively on this instance until all keys in key_tuple are exhausted.

Setting suppress_missing to True will internally open the Settings.suppress_missing() context manager, thus raising a KeyError if a key in key_tuple is absent from this instance.

>>> s = Settings()
>>> s.a.b.c = True
>>> value = s.get_nested(('a', 'b', 'c'))
>>> print(value)
True

set_nested(key_tuple, value, suppress_missing=False)[source]¶

Set a nested value by, recursively, iterating through this instance using the keys in key_tuple.

The Settings.__getitem__() method is called recursively on this instance, followed by Settings.__setitem__(), until all keys in key_tuple are exhausted.

Setting suppress_missing to True will internally open the Settings.suppress_missing() context manager, thus raising a KeyError if a key in key_tuple is absent from this instance.

>>> s = Settings()
>>> s.set_nested(('a', 'b', 'c'), True)
>>> print(s)
a:
  b:
    c:      True

flatten(flatten_list=True)[source]¶

Return a flattened copy of this instance.

New keys are constructed by concatenating the (nested) keys of this instance into tuples.

Opposite of the Settings.unflatten() method.

If flatten_list is True, all nested lists will be flattened as well. Dictionary keys are replaced with list indices in such case.

>>> s = Settings()
>>> s.a.b.c = True
>>> print(s)
a:
  b:
    c:      True

>>> s_flat = s.flatten()
>>> print(s_flat)
('a', 'b', 'c'):    True

unflatten(unflatten_list=True)[source]¶

Return a nested copy of this instance.

New keys are constructed by expanding the keys of this instance (e.g. tuples) into new nested Settings instances.

If unflatten_list is True, integers will be interpretted as list indices and are used for creating nested lists.

Opposite of the Settings.flatten() method.

>>> s = Settings()
>>> s[('a', 'b', 'c')] = True
>>> print(s)
('a', 'b', 'c'):    True

>>> s_nested = s.unflatten()
>>> print(s_nested)
a:
  b:
    c:      True

__iter__()[source]¶

Iteration through keys follows lexicographical order. All keys are sorted as if they were strings.

__missing__(name)[source]¶

When requested key is not present, add it with an empty Settings instance as a value.

This method is essential for automatic insertions in deeper levels. Without it things like:

>>> s = Settings()
>>> s.a.b.c = 12

will not work.

The behaviour of this method can be suppressed by initializing the Settings.suppress_missing context manager.

__contains__(name)[source]¶

Like regular __contains`__, but ignore the case.

__getitem__(name)[source]¶

Like regular __getitem__, but ignore the case.

__setitem__(name, value)[source]¶

Like regular __setitem__, but ignore the case and if the value is a dict, convert it to Settings.

__delitem__(name)[source]¶

Like regular __detitem__, but ignore the case.

__getattr__(name)[source]¶

If name is not a magic method, redirect it to __getattribute__.

__setattr__(name, value)[source]¶

If name is not a magic method, redirect it to __setattr__.

__delattr__(name)[source]¶

If name is not a magic method, redirect it to __delattr__.

_str(indent)[source]¶

Print contents with indent spaces of indentation. Recursively used for printing nested Settings instances with proper indentation.

__str__()[source]¶

Return str(self).

__dir__()[source]¶

Return standard attributes, plus dynamically added keys which can be accessed via dot notation.

__repr__()¶

Return repr(self).

Global settings¶

Global settings are stored in a public ConfigSettings instance named config. They contain variables adjusting general behavior of PLAMS as well as default settings for various objects (jobs, job manager etc.)

The default values are explained in the description of each property on config. It is recommended to have a look at that these options, to give an overview of what behaviour can be configured.

To change a setting for a script, just set the relevant option on the config to the preferred value, after the import statements. For example:

config.log.stdout = 1
config.job.pickle = False
config.default_jobrunner = JobRunner(parallel=True, maxjobs=8)

The structure for the defined options on the nested settings objects are defined below.

class ConfigSettings(*args, **kwargs)[source]¶

Extends the default Settings with standard options which are required for global config. The values for these options are initialised to default values. The default JobRunner and JobManager are lazily initialised when first accessed.

__init__(*args, **kwargs)[source]¶

property init¶

Whether config has been marked as fully initialized and jobs are ready to be run. Defaults to False.

property _explicit_init¶

Whether config has been explicitly initialized by the user by calling init(). Defaults to False.

property preview¶

When enabled, no actual calculations are run, only inputs and runscripts are prepared. Defaults to False.

property sleepstep¶

Unit of time which is used whenever some action needs to be repeated until a certain condition is met. Defaults to 5.

property ignore_failure¶

When enabled, accessing a failed/crashed job gives a log message instead of an error. Defaults to True.

property daemon_threads¶

When enabled, all threads started by JobRunner are daemon threads, which are terminated when the main thread finishes, and hence allow immediate end of the parallel script when Ctrl-C is pressed. Defaults to True.

property erase_workdir¶

When enabled, the entire main working folder is deleted at the end of script. Defaults to False. :return:

property jobmanager¶

See JobManagerSettings.

property job¶

See JobSettings.

property log¶

See LogSettings.

property saferun¶

See SafeRunSettings.

property default_jobrunner¶

Default JobRunner that will be used for running jobs, if an explicit runner is not provided.

property default_jobmanager¶

Default JobManager that will be used when running jobs, if an explicit manager is not provided.

class JobSettings(*args, **kwargs)[source]¶

Job settings for global config.

__init__(*args, **kwargs)[source]¶

property pickle¶

Enables pickle for the whole job object to [jobname].dill, after job execution is finished. Defaults to True.

property pickle_protocol¶

Protocol used for pickling. Defaults to -1.

property keep¶

Defines which files should be kept on disk. Defaults to all.

property save¶

Defines which files should be kept on disk. Defaults to all.

property runscript¶

See RunscriptSettings.

property link_files¶

When enabled, re-run files will be hardlinked instead of copied, unless on Windows when files are always copied. Defaults to True.

class JobManagerSettings(*args, **kwargs)[source]¶

Job manager settings for global config.

__init__(*args, **kwargs)[source]¶

property counter_len¶

Number of digits for the counter used when two or more jobs have the same name, when all the jobs apart from the first one are renamed to [jobname].002 ([jobname].003 etc.) Defaults to 3.

property hashing¶

Hashing method used for testing if some job was previously run. Defaults to input.

property remove_empty_directories¶

When enabled, removes of all empty subdirectories in the main working folder at the end of the script. Defaults to True.

class LogSettings(*args, **kwargs)[source]¶

Log settings for global config.

__init__(*args, **kwargs)[source]¶

property file¶

Verbosity of the log printed to .log file in the main working folder. Defaults to 5.

property stdout¶

Verbosity of the log printed to the standard output. Defaults to 3.

property time¶

When enabled, include write time for each log event. Defaults to True.

property date¶

When enabled, include write date for each log event. Defaults to True.

class RunScriptSettings(*args, **kwargs)[source]¶

Run script settings for global config.

__init__(*args, **kwargs)[source]¶

property shebang¶

First line of all produced runscripts. Defaults to #!/bin/sh.

property stdout_redirect¶

When enabled, the standard output redirection is handled by the operating system (by using ‘>[jobname].out’ in the runscript), instead of being handled by native Python mechanism. Defaults to False.

class SafeRunSettings(*args, **kwargs)[source]¶

Safe run settings for global config.

__init__(*args, **kwargs)[source]¶

property repeat¶

Number of attempts for each run() call. Defaults to 10.

property delay¶

Delay between attempts for each run() call. Defaults to 1.