5.1. Benchmarks from literature

This tutorial shows you how you can

  • Use any AMS engine(s) to evaluate a dataset and compare to reference values

../../_images/drawing.png

Fig. 5.1 An example reaction energy from the ISOL benchmark. The predictions were evaluated with DFT (PBE/TZP). This tutorial will instead use UFF, which gives much worse predictions.

Here, ParAMS is not used for parametrization but simply as a convenient tool to run multiple calculations and compare the results to reference values.

This is accomplished by

  • running a SinglePointPanel calculation (SinglePoint documentation)

  • using the NoParameters parameter interface (Parameters → No Parameters)

  • setting the ParAMS Engine settings

We will use ParAMS together with an AMS engine to evaluate a subset of the ISOL isomerization benchmark set. The reference energies were evaluated at the CCSD(T)-F12a/aDZ level of theory.

The example files are stored in $AMSHOME/scripting/scm/params/examples/benchmark.

5.1.1. Create the reference values

The reference values and corresponding jobs need to follow the ParAMS format (job_collection.yaml, training_set.yaml, etc.)

Note

The reference values are stored in training_set.yaml, even though no “training” will take place.

In the example directory $AMSHOME/scripting/scm/params/examples/benchmark all files that you need for this tutorial have already been created. The below tabs give some hints on how you can create your own files.

If you want to use ParAMS and AMS to create your reference data, see the Generate reference values tutorial.

This will automatically create the data in a format that ParAMS can read.

5.1.2. Set the engine settings

Start the ParAMS GUI
File → Open
Open $AMSHOME/scripting/scm/params/examples/benchmark/job_collection.yaml
Parameters → No Parameters
Switch to the Training Set panel

There are five reaction energies in the data set.

Select one of the lines in the table
Use the left and right arrow keys on your keyboard to toggle between the structures in the molecule view.
Switch to the Jobs panel

The ParAMS Engine for all jobs is set to UFF. This means that all jobs will be evaluated with the engine whose ID is UFF.

Switch to the Engines panel

This tutorial comes with a few different engines pre-defined. UFF is one of them.

Double-click on the UFF engine

This brings up the Engine settings that will be used during the calculation. To modify the settings for the UFF engine, either

  • Type the settings directly in the text box

  • Use the Edit in AMSInput button

In this case, there is no need to make any changes.

Click the Cancel button

If you want, you can change the engine for all the jobs:

Switch to the Jobs panel
Select the top job
Hold Shift and click on the bottom job to select all jobs
Double-click in the Details column for one of the jobs
Switch the ParAMS Engine ID to for example ReaxFF:CHON-2019
Click OK

The rest of this tutorial uses the UFF ParAMS engine.

5.1.3. Run the SinglePoint job

Click on the SinglePointPanel panel

This brings up the SinglePoint settings. Here, we will just use the default settings.

File → Save as
Switch to AMSjobs: SCM → Jobs
Select the job and run

5.1.4. Results from the SinglePoint calculation

The results from a SinglePoint are similar to those from an Optimization.

Switch to the Graphs panel

This shows a scatter plot of predicted vs. reference energies.

../../_images/benchmark_graphs_tab.png
Switch to the Results panel

Here, you see for example that the mean absolute error (MAE) for the predicted isomerization energies is 33.4 kcal/mol.

../../_images/benchmark_results_tab.png

5.1.5. More SinglePoint possibilities

  • Use different engine settings for different jobs (e.g., different spin polarizations or k-space samplings)

  • If your SinglePoint job exits before all jobs have finished, you can restart using the Load jobs from option in the SinglePointPanel tab.

  • To change from one engine to the other, it is easiest to just open a text editor and do a search-and-replace on ExtraEngineID: UFF to for example ExtraEngineID: DFTB in job_collection.yaml.