SCF troubleshooting

SCF convergence problems can have various reasons. Thus, finding the reasons for a particular SCF behavior is a half-way to solving the problem. You’ll be surprised but the majority of SCF convergence problems are caused by a non-physical calculation setup, such as mistakes in the geometry or a too large negative charge. Thus, the first thing to do is to check if the geometry is really what it is meant to be. Check for too short inter-atomic distances, make sure the coordinates are specified in the right units. By default ADF expects atomic coordinates in Angstrom so check that the coordinates are provided in these units. Also check that no atoms got “lost” when importing coordinates.

So, your calculation is set up correctly, but the SCF still does not converge. Before discussing other options, let’s look at how the SCF process in ADF is organized. In a nutshell, it consists of the following steps:

  • A Fock matrix is constructed from the current density and the potential.
  • The Fock matrix is used in a DIIS procedure where it is mixed with some previous Fock matrices to construct a new one. By default ADF uses a mixed ADIIS+DIIS scheme.
  • This new Fock matrix is diagonalized to obtain molecular orbitals (MOs).
  • MOs are populated by electrons following the aufbau principle or, if the KeepOrbitals feature is ON, by overlap with a previous density matrix.
  • A new density matrix is constructed from occupied MOs and the steps 1-5 are repeated.

Problems on any of the steps 1, 2, and 4 above can cause problems in the whole SCF process. Usually one can identify which step causes the problems by looking in the logfile and in the output file. In the logfile, two values are printed for each SCF cycle: ErrMat and MaxEl. Both values are related to commutator of the current Fock and density matrices, [F,P]. ErrMax is a sum of squares of the commutator matrix elements while MaxEl is its largest (by absolute value) element. Below, different SCF patterns will be discussed with suggestions on how to solve them.

By far the most common reason for non-converging SCF is a very small or absent HOMO-LUMO gap. This problem is most frequently observed for compounds containing d- and f-elements (transition and rare-earth metals). This causes different MOs to be populated in subsequent cycles at the step 4 above, which, in turn, leads to large changes in the density and Fock matrices between cycles. In the logfile, the problem manifests itself by the ErrMat and MaxEl values remaining rather large (in the order of 0.1 to a few tens) sometimes going down to smaller values but then jumping back up. By looking at the MO population numbers in the output file one can sometimes see that the HOMO changes from cycle to cycle. There are a few ways to get the SCF converged in such a situation.

For open-shell electronic configurations, it is possible that a spin-unrestricted calculation will converge better than a spin-restricted one. Thus, if the molecule is not going to be used as a fragment (in which case it must be spin-restricted) then it is recommended to perform a spin-unrestricted calculation in a high-spin configuration. This is particularly useful for molecules with multiple radical centers, such bi- or multi-nuclear transition metal complexes. After a high-spin calculation of the complex has converged one can perform a broken-symmetry low-spin calculation using the high-spin results as a restart and a SpinFlip feature. See also a tutorial on spin coupling of an iron compound for an example of this approach.

If a spin-unrestricted calculation is not desirable or if it also has SCF convergence problems one may consider trying different DIIS methods in the step 2 above or try a completely different SCF method, preferably in the order listed below:

  • A-DIIS is used by default but it is incompatible with non-aufbau electronic configurations so it may be disabled in this case and SDIIS is used. A non-aufbau electronic configuration may be enforced using a block form of the Occupations key, but it may also result from the KeepOrbitals (a.k.a. orbital tracking) feature, or by the Fermi orbital population scheme. Thus, check if A-DIIS has been disabled (i.e. you do not see any A-DIIS coefficients in the SCF part of the text output) and if it was then you can try using one of the options below.
  • The LIST family of methods are developed by Y.A. Wang and coworkers. The convergence when using fDIIS, LISTb, LISTf, or LISTi is often comparable to that of A-DIIS, but sometimes it may be better or worse. Like A-DIIS, these methods also replace the DIIS step only leaving other steps unchanged. In the LIST methods, the number of the expansion vectors (also known as the size of the DIIS space) is an important factor which may have to be adjusted to achieve convergence.
  • The MESA method is developed by the authors of the LIST family. This method combines up to five individual acceleration methods. From our experience, using this method in combination with the Fermi orbital population one can converge transition metal clusters in the shortest time even though each SCF cycle is somewhat more expensive when using MESA than when using any of the acceleration methods mentioned above.
  • Energy-DIIS by Scuseria and Kudin is a good method but in general not better than the LIST family methods. It requires evaluation of the total energy, which is its strong point and a weakness at the same time. It is a strong point because it lets the algorithm converge the SCF to a configuration with the lowest energy and it is a weakness because energy evaluation is a computationally expensive procedure in ADF. To use Energy-DIIS just add an EDIIS keyword to the SCF input block. Energy-DIIS can only be used in combination with OldSCF.
  • The Augmented Roothaan-Hall (ARH) method is an alternative SCF method that combines steps 2-4 above into a single step. Essentially, ARH performs a direct minimization of the energy as a function of the density matrix combining a preconditioned conjugate-gradient method with a trust-radius approach. This is probably the most powerful SCF method to date because it can converge even the most difficult cases. However, it also has its limitations and drawbacks discussed in the corresponding section of the ADF User’s Guide. The most important drawback is that, like Energy-DIIS, it also requires evaluation of the total energy. Besides, for the method to work reliably, the energy must be accurate, which means that an accurate fit and high integration accuracy should be used. ARH can only be used in combination with OldSCF.

As mentioned above, A-DIIS, LIST and Energy-DIIS affect only the DIIS step of the whole SCF process. Thus, it is possible that the SCF still has trouble converging even with the best DIIS method, because different MOs are occupied in different SCF cycles, which induces large changes in the density. ADF has a built-in feature called KeepOrbitals that assigns electrons to MOs based on their overlap with occupied MOs from the previous SCF cycle. KeepOrbitals is usually switched on after half of the allowed SCF cycles has passed (150 by default). However, if there are SCF problems, switching on KeepOrbitals may not have the desired effect. For example, the system may be trapped in an excited state due to it. Changing the SCF cycle at which KeepOrbitals kicks in may affect the final electronic configuration. Thus, playing with KeepOrbitals and trying different values for its parameter (the SCF cycle number) is encouraged.

For the DIIS-like methods listed above, the number of expansion vectors is an important parameter. We found that in some cases, for example, transition metal clusters, increasing this number to 20 or 25 solves most of the problems. Thus, the first thing to try when there are SCF convergence problems is to use the following input parameters:

SCF
  AccelerationMethod ADIIS
  DIIS N=20
END
Occupations KeepOrbitals=300

The input above uses the default A-DIIS but increases the number of expansion vectors to 20 and disables KeepOrbitals during the whole SCF process. You can also try replacing ADIIS with any method from the LIST family.

If none of the individual acceleration methods work one can try using MESA with the increased number of DIIS vectors:

SCF
  AccelerationMethod MESA
  DIIS N=20
END

Numerical noise in the exchange-correlation potential may also contribute to SCF convergence problems. If this is the case, the SCF process starts off converging quite well until some point after which the ErrMat and MaxEl values remain relatively small but do not decrease further. This behavior is typically observed for systems where weak (Van der Waals or hydrogen bonding) interactions are present. It is caused by the density fit not being good enough in the chemically relevant region between weakly bonded fragments.

This problem can be resolved by adding the EXACTDENSITY keyword to the input file. When using ADFinput, the corresponding option called “Density used in XC-potential” found on the Accuracy tab should be set to “Exact” or “Exact MO-based”, “Exact” being preferred.