Installation¶

Floating license¶

Note: floating licenses are not supported on Windows.

If you have a floating license, you will need to follow these instructions below to set it up. The instructions are simple, but it is important you follow them exactly.

If you do not have a floating license you may skip this section.

Create a floating license directory

Make a new directory, for example FloatADF, to keep track of the running ADF processes.

This FloatADF directory must be:

• in a fixed location (the directory name should not change!)
• shared between / mounted on all nodes where you want to run ADF
• writable by all users that want to run ADF
• FloatADF must not be a subdirectory of $ADFHOME

For example:

cd /usr/share
mkdir FloatADF
chmod 1777 FloatADF

If you also have a floating license for other modules, you need to set up different directories and repeat this procedure for all these modules (for example FloatBAND, FloatReaxFF, FloatDFTB, ...)

In the example, we have given all users read, write and execute permissions to this directory. If you wish to restrict this for security reasons, you may do so as long as all ADF users will have read, write and search permission for this directory. You may create an ADF user group for this purpose.

Important: the directory should not be a sub-directory of $ADFHOME as the directory name will change if you update to a new version! Also, do not put the license.txt file in the FloatADF directory.

The FloatADF directory may not be moved, deleted or renamed for the duration of your license because these actions will invalidate it!

E-mail us the license information Send the result of the following commands (using again the name FloatADF and the location /usr/share as an example) to license@scm.com:

cd /usr/share
ls -lid $PWD/FloatADF

Please note that output of the ls command must include the full path of the FloatADF directory.

Together with the output of the ls command above, you also need to send us output of the $ADFBIN/dirac info command from each computer on which ADF will possibly run, as the license file is still host-locked.

In the case of very large clusters, it is often sufficient if you send us the output of the $ADFBIN/dirac info command from the head node and 2 or 3 compute nodes. As most compute node names have the same beginning, we can then add them with a wild card (for example Linuxchem*). It is important when you use this facility, to let us know that the info you are sending are not all compute nodes. Otherwise the license will only run on these few compute nodes.

Check the license file¶

Windows users test their license by double-clicking the makelicinfo.bat file in the ADF installation folder.

Unix users: first check that the license file has been installed correctly by running (at the shell prompt):

$ADFBIN/dirac check

This should produce the output similar to the following:

Checked: /home/testadf/adf2017.xxx/license.txt

 License termination date (mm/dd/yyyy):  1/ 8/2017

 According to it, you are allowed to run:

           ADF version  2017.990
        ADFGUI version  2017.990
           GUI version  2017.990
     REAXFFGUI version  2017.990
          BAND version  2017.990
       BANDGUI version  2017.990
           CRS version  2017.990
        DCDFTB version  2017.101
          DFTB version  2017.990
       DFTBGUI version  2017.990
         MOPAC version  2017.990
           NBO version  2017.990
          NBO6 version     6.000
        QNDFTB version  2017.990
        REAXFF version  2017.990
         Utils version  2017.990

 Number of procs you are allowed to use for:

 ADF    :    1024 procs
 BAND   :    1024 procs
 DFTB   :    1024 procs
 ReaxFF :    1024 procs

 =====================================================
 SCM User ID: u999999
 release: 2017.101
 :example.com:
 :Linuxmaster.example.com00:11:22:33:44:55:
 :ncores      12:
 :CPU Model Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz:
 :DMY  1- 7-2016:
 :SRCID  3217288:
 =====================================================
LICENSE INFO READY

If the license check is successful (i.e. you get the “LICENCE INFO READY” message) you can move on. Otherwise check that the license file has been installed according to instructions above.

Run some basic tests¶

Verify that you can now run examples provided with ADF and that they give correct results. We recommend that you consult the Examples document for notes on such comparisons: non-negligible differences do not necessarily indicate an error. If you have a license for the GUI you can also run a few GUI tutorials as a test.

Note: the example .run files are complete Bourne shell scripts that should be executed as such, they are **not* input files to be fed into any program. The easiest way to run them is using ADFjobs.

Test the GUI¶

If the GUI is included in your license, check that you can start any of the GUI modules.

UNIX users:

Enter the following command:

$ADFBIN/adfjobs

An ADFjobs window should appear.

Mac users:

Double click the ADF2017.xxx application to start it. An ADFjobs window should appear.

Windows users:

Double click the ADFjobs icon to start it. An ADFjobs window should appear. If the window does not appear or appears after a long delay then check the ADF-GUI requirements and check the firewall rules that should allow local communication.

All users should now be able to start ADFinput via the SCM menu on the top left of the menu bar.

Test parallel execution¶

It is very important to make sure that computer resources are utilized with the maximum efficiency. Therefore, you should check that each ADF job uses all processors/cores allocated to it and that the network is not overloaded with the disk I/O traffic.

Typically, when you submit a parallel job to the batch system you have a possibility to specify how many processors per node (ppn) to use. If the batch system you are using allows this, then make sure that you request ppn equal to the number of physical cores on each node. Note that recent AMD processors from the Familiy 15h based on the so-called “Bulldozer” cores have one floating-point unit (module) per two physical cores. On these Bulldozer architectures and its successors (Piledriver, Steamroller) ADF will probably perform best when you only run on half the physical cores. ADF tries to automatically detect the number of floating-point units (half the physical cores), but the user is advised to check this and otherwise set ppn accordingly.

It is also possible that your batch system does not allow you to specify ppn but instead it always assumes that there only one processor per node. In this case, you will need to edit the $ADFBIN/start file and add some commands for processing the hostfile.

In order to check that everything is working as intended, create a reasonably small ADF job and start it, preferrably on more than one node. After the job has finished, open the job’s .out file and find the table that looks like the following:

Parallel Execution: Process Information
==============================================================================
Rank   Node Name                              NodeID   MyNodeRank  NodeMaster
   0   compute-0-0                               0          0          0
   1   compute-0-0                               0          1         -1
   2   compute-0-0                               0          2         -1
   3   compute-0-0                               0          3         -1
   4   compute-0-1                               1          0          1
   5   compute-0-1                               1          1         -1
   6   compute-0-1                               1          2         -1
   7   compute-0-1                               1          3         -1
==============================================================================

Check the names in the “Node Name” column and verify that the number of tasks per node is correct. If it is, then move on to the next section.

Test parallel performance¶

If the process allocation to nodes looks as expected then scroll down a bit to the following lines:

Communication costs MPI_COMM_WORLD:      1.026 usec per message,   0.0320 usec per 8-byte item
Communication costs for intra-node:      1.023 usec per message,   0.0318 usec per 8-byte item

Make sure that you get reasonable numbers (less than 100 usec per message) both for intra-node and MPI_COMM_WORLD communication costs. Otherwise contact your system administrator. If only the MPI_COMM_WORLD value is large but the intra-node one looks reasonable, this may mean that the network link between machines is very slow and you may want to run single-node jobs only.

If all seems to be OK then move to the end of the file to the table called “Timing Statistics”. Before the table, there is a line of text beginning with “Total Used” that might look as follows:

Total Used :   CPU=     8064.87      System=     1144.31     Elapsed=     9212.99

This shows how much time (in seconds) was spent in the ADF code (CPU) and in the kernel (System), and how long did it take for the calculation to complete (Elapsed). Ideally, the system time should be small compared to the CPU time and the latter should be close to the Elapsed time. The system time will not always be a small number, however, the sum of the System and CPU times should always give a number very close to the Elapsed time. If this is not the case then it means that ADF has to spend a lot of time waiting for something. This can be, for example, disk I/O or network communication.

If you notice that the system time portion is enormously large or that there is a large difference between the CPU+System and the Elapsed time, then repeat the job with the “PRINT TimingDetail” keyword in the input and contact the SCM support.

ADFjobs queues¶

A video shows how to set up remote queues on Windows, other platforms are even easier.

ADFjobs can submit and monitor jobs for you on different queues, both locally and remotely. You can use the GUI on your desktop or laptop for creating, submitting and analyzing your jobs, while running the jobs on remote compute clusters. In that case you should establish and ssh-agent connection, to enable remote job managing by the GUI. If you just run local jobs without a queuing system, skip this section.

To set up a new batch queue, select Queue → New... → and choose an example queuing system to starting with (LSF, PBS, or SGE). Modify the input files such that they correspond to your queue configuration:

• change the Name of the queue
• set the Remote host to the hostname of your cluster
• set Remote user to your username on that cluster
• change the Run command in accordance with your typical queue set up. The $options corresponds to the input box that can be modified in ADFjobs before submitting, e.g. the time or number of nodes
• you can set what $options defaults to in the Default Options field
• change the Kill, Job status, System status commands, if necessary
• you may define a Prolog or Epilog command

Managing remote jobs¶

To manage remote jobs, you need automatic authentication with an ssh-agent via an ssh key pair. This is most easily set up on MacOS or Linux, but is a bit more involved for Windows.

ssh connection on MacOS and Linux

If you don’t have an ssh key pair already, you can generate one in a terminal: ssh-keygen -t rsa Put your pass-word protected public key on the compute cluster(s) in ~/.ssh/authorized_keys.

Next, you should establish a connection via an ssh-agent. On MacOS an ssh-agent runs by default, and with keychain you just have to type your password once when you make the first connection.

On various Linux installations ssh-agent is also running by default. If an ssh-agent daemon is not yet running in the terminal where you will run ADFjobs, start the daemon, e.g. by adding eval $(ssh-agent bash) to your .bashrc. You need to add your private key to the ssh-agent daemon: ssh-add.

ADFjobs will use SSH to connect to the remote systems, making a new connection for everything it does. This means that there will be many connections to the remove machine(s). If you arre using OpenSSH (both on the local and the remote machine), you can instead use one ssh connection and keep it alive, reusing it for many things. This makes ADFjobs faster, and may avoid complaining system administrators of the remote machines.

To do this, set the environment variable SCM_SSH_MULTIPLEXING to yes (for example via the GUIprefs aplication).

ssh connection on Windows

PuTTY tools, automatically installed with ADF, can be used to set up an ssh-agent connection. Follow these steps, using the PuTTY programs located in the binPutty directory in your ADF installation directory.

• Run puttygen.exe and follow the instructions to generate an ssh key pair. Make sure you use a password. Put the public key on the remote host in ~/.ssh/authorized_keys
• Run pageant.exe and add your private ssh key (right-click on the Pageant icon in the task bar) with your password
• Open a Command Prompt and go the Putty directory (e.g. cd C:ADF2017.xxxbinPutty). Run plink user@host (with correct username and hostname) and type y when prompted to store the key in cache
• In the same Command Prompt, check that you can now connect through the ssh-agent: plink -batch -agent -l user host uptime

Now you can close the prompt and start using ADFjobs to manage remote jobs. You may test your queue configuration: Jobs → Generate Test Job and assign it to your new queue before running the test job.

Centrally defined queues

A system administrator may also define a queue centrally, which may then be picked up by any user automatically via Dynamic queues. Consult the GUI manual for information on how to set these up in ADFjobs.

MOPAC

MOPAC is included with the ADF distribution. However, it needs to be enabled in your license file. If it is not enabled, please contact SCM for more information. Note that MOPAC is free for academic users but the MOPAC key has a limited duration. The MOPAC included with the ADF distribution is just the standard MOPAC from OpenMOPAC, and is updated frequently.

If you wish to use a MOPAC version different from the one included with the ADF distribution, you can do this by setting the SCM_MOPAC environment variable, either in your shell startup script or via the SCM → Preferences command:

• do not set SCM_MOPAC when you want to run the MOPAC included with the ADF pacakage, in most situations this is the easiest solution
• set SCM_MOPAC to the complete path to the Mopac executable
• set SCM_MOPAC to a command if you want to run MOPAC on a different machine, the command must pass the arguments and standard input, and should start the mopac.scm script on the other machine (located in $ADFBIN on that machine)

ffmpeg

The ADFmovie GUI module can make MPEG videos of the system as you see it in the ADFmovie window. This can be an MD trajectory or vibrational modes, or the course of geometry optimization. For this to work, you need to install the free ffmpeg program from the FFmpeg website and make sure it can be found in the path. The simplest way to do this is to copy the ffmpeg executable to $ADFBIN.