6.4. Task: Sensitivity¶
Task Sensitivity runs an HSIC sensitivity analysis quantifying the effect of active parameters on the loss function.
To understand this task better, see the tutorial on Parameter sensitivity analysis.
6.4.1. Generating Samples¶
The first step of the sensitivity analysis is obtaining i.i.d uniformly-distributed random samples of the parameter space.
Samples can be automatically generated as part of the sensitivity calculation by using the RunSampling key:
RunSampling- Type
Bool
- Default value
No
- Description
Produce a set of samples of the loss function and active parameters. Samples from the parameter space are drawn from a uniform random distribution. Such a set of samples serves as the input to the sensitivity calculation.
Note
If you chose to generate samples, then the shared collection keys must also be defined so that ParAMS can construct the loss function to sample.
If you generating samples, you may select the number and manner in which you would like to sample:
NumberSamples- Type
Integer
- Default value
1000
- GUI name
Generate n samples:
- Description
Number of samples to generate during the sampling procedure.
RandomSeed- Type
Integer
- Description
Random seed to use during the sampling procedure (for reproducibility).
SaveResiduals- Type
Bool
- Default value
No
- Description
During the sampling, save the individual difference between reference and predicted values for every sample and training set item. Required for the Reweight calculation, and will be automatically activated if the reweight calculation is requested. Saving and analyzing the residuals can provide valuable insight into your training set, but can quickly occupy a large amount of disk space. Only save the residuals if you would like to run the reweight calculation or have a particular reason to do so.
Tip
You can reuse the residuals to calculate new loss values:
with different loss functions;
after changes to weights/sigmas; and
after removing training set items.
This allows you to tweak and tailor your training set and run a new sensitivity calculation without having to resample.
6.4.2. Loading Samples¶
Instead of generating samples, they can be loaded:
SamplesDirectory- Type
String
- Default value
- Description
Path to an ‘optimization’ directory containing the results of a previously run sampling. First looks for a ‘glompo_log.h5’ file. If not found, will look for ‘running_loss.txt’ and ‘running_active_parameters.txt’ in a sub-directory. The sub-directory used will depend on the DataSet Name. For the Reweight calculation only a ‘glompo_log.h5’ file (with residuals) may be used.
6.4.3. Kernels¶
The most important configurable options are the kernels applied to the parameter values and loss values.
We generally recommend:
applying the Gaussian kernel to the parameter values (in order to capture dis/similarity between parameter sets);
applying the conjunctive-Gaussian kernel to the loss values (in order to focus the weight of the distribution on good minima); and
using polynomial or linear kernels only if you have a specific type of relationship/dependency you would like to investigate.
ParametersKernel
Alpha float
Gamma float
Polynomial
Order integer
Shift float
End
Sigma float
Type [Gaussian | ConjunctiveGaussian | Threshold | Polynomial | Linear]
End
ParametersKernel- Type
Block
- Description
Kernel applied to the parameters for which sensitivity is being measured.
Alpha- Type
Float
- Description
Cut-off parameter for the Threshold kernel between zero and one. All loss values are scaled by taking the logarithm and then adjusted to a range between zero and one. This parameter is a value within this scaled space.
Gamma- Type
Float
- Default value
0.1
- Description
Bandwidth parameter for the conjunctive-Gaussian kernel.
Polynomial- Type
Block
- Description
Settings for the Polynomial kernel.
Order- Type
Integer
- Default value
1
- Description
Maximum order of the polynomial.
Shift- Type
Float
- Default value
0.0
- Description
Free parameter (≥ 0) trading off higher-order versus lower-order effects.
Sigma- Type
Float
- Default value
0.3
- Description
Bandwidth parameter for the Gaussian kernel.
Type- Type
Multiple Choice
- Default value
Gaussian
- Options
[Gaussian, ConjunctiveGaussian, Threshold, Polynomial, Linear]
- Description
Name of the kernel to applied to the parameters for which sensitivity is being measured.
LossValuesKernel- Type
Block
- Description
Kernel applied to the parameters for which sensitivity is being measured.
Alpha- Type
Float
- Description
Cut-off parameter for the Threshold kernel between zero and one. All loss values are scaled by taking the logarithm and then adjusted to a range between zero and one. This parameter is a value within this scaled space.
Gamma- Type
Float
- Default value
0.3
- Description
Bandwidth parameter for the conjunctive-Gaussian kernel.
Polynomial- Type
Block
- Description
Settings for the Polynomial kernel.
Order- Type
Integer
- Default value
1
- Description
Maximum order of the polynomial.
Shift- Type
Float
- Default value
0.0
- Description
Free parameter (≥ 0) trading off higher-order versus lower-order effects.
Sigma- Type
Float
- Description
Bandwidth parameter for the Gaussian kernel. If not specified or -1, calculates a reasonable default based on the number of parameters being tested.
Type- Type
Multiple Choice
- Default value
ConjunctiveGaussian
- Options
[Gaussian, ConjunctiveGaussian, Threshold, Polynomial, Linear]
- Description
Name of the kernel to applied to the parameters for which sensitivity is being measured.
6.4.4. Other settings¶
The sensitivity analysis can be run on either the training set or the validation set:
SetToAnalyze- Type
Multiple Choice
- Default value
TrainingSet
- Options
[TrainingSet, ValidationSet]
- GUI name
Analyze:
- Description
Name of the data set to use for the sensitivity analysis.
Generally, all of the samples generated are not used simultaneously in the sensitivity calculation. This is because the more samples used, the slower the calculation. It is also difficult to know if one has enough samples to capture an accurate approximation of the true sensitivity.
Therefore, suppose one has generated 10000 samples. It is often better to run 10 repeats (bootstraps) of the calculation with 1000 points each than it is to run a single calculation with 10000 points. The first configuration will calculate faster, and provide a spread of the data to evaluate the robustness of the result.
To specify the number of times you would like the calculation repeated:
NumberBootstraps- Type
Integer
- Default value
1
- GUI name
Repeat calculation n times:
- Description
Number of repeats of the calculation with different sub-samples. A small spread from a large number of bootstraps provides confidence on the estimation of the sensitivity.
To specify how many points to use from our sample set in each calculation:
NumberCalculationSamples- Type
Integer
- GUI name
Number of samples per repeat:
- Description
Number of samples from the full set available to use in the calculation. If not specified or -1, uses all available points. For the sensitivity calculation, this will be redrawn for every bootstrap.
To specify how the points are chosen from our sample set:
SampleWithReplacement- Type
Bool
- Default value
Yes
- Description
Sample from the available data with or without replacement. This only has an effect if the number of samples for the calculation is less than the total number available otherwise replace is Yes by necessity.
To remove parameter sets which produced non-finite loss values:
FilterInfiniteValues- Type
Bool
- Default value
Yes
- Description
If Yes, removes points from the calculation with non-finite loss values. Non-finite points can cause numerical issues in the sensitivity calculation.
Finally, we have included an extension to the sensitivity calculation:
RunReweightCalculation- Type
Bool
- Default value
No
- Description
Run a more expensive sensitivity calculation that will also return suggested weights for the training set which will produce more balanced sensitivities between all the parameters. Note: The Gaussian kernel is recommended for the loss values kernel in this case.
Warning
The reweight calculation is experimental.
6.4.5. Technical details¶
For technical details of the sensitivity calculation see the API documentation