Quality metrics#
mergen.metrics#
Design quality metrics and optional Monte Carlo baseline comparison.
Usage#
result.quality_report() result.quality_report(metrics=[‘min_distance’, ‘cd2’]) result.quality_report(mc_samples=500) result.quality_report(criteria_metrics=[‘maxpro’, ‘phi_p’], mc_samples=500)
Metrics (always computed from the design, no MC needed)#
min_distance Johnson et al. (1990) — worst-case separation minimax Johnson et al. (1990) — fill distance / coverage max_abs_correlation — orthogonality between parameters projection_cd2 Liu & Liu (2023) — 2D projection uniformity cv_distances Lekivetz & Jones (2015) — clustering index mean_distance — average pairwise distance
Plus the criterion used in sampler.run() is always included.
criteria_metrics (optional, computed from the design)#
Any SA criterion name: ‘umaxpro’, ‘maxpro’, ‘phi_p’, ‘cd2’, ‘stratified’ Useful when the user wants to evaluate the design under a different criterion than the one used for optimisation.
MC baseline (optional, mc_samples > 0)#
Draws mc_samples random designs from the feasible space. For each metric and criteria_metric, computes:
baseline median
percentile rank of the design value among random designs
References
Johnson, Moore & Ylvisaker (1990), J. Statist. Plan. Infer. 26(2). Hickernell (1998), Math. Comp. 67. Lekivetz & Jones (2015), Quality Engineering 27(1). Liu & Liu (2023), Canadian Journal of Statistics 51(1).
- mergen.metrics.metric_latex(name)[source]#
Return the LaTeX math label for a metric, or the name itself.
- mergen.metrics.min_distance(X, space=None)[source]#
Minimum normalised pairwise distance (maximin criterion).
Higher values indicate better point separation. When space contains nominal factors the pairwise distance is HEOM (Wilson & Martinez 1997); otherwise the standard normalised Euclidean distance is used.
References
- Johnson, M. E., Moore, L. M. & Ylvisaker, D. (1990).
Journal of Statistical Planning and Inference, 26(2), 131-148.
- Wilson, D. R. & Martinez, T. R. (1997). Improved heterogeneous
distance functions. J. Artif. Intell. Res., 6, 1-34.
- mergen.metrics.mean_distance(X, space=None)[source]#
Mean normalised pairwise distance.
Higher values indicate better overall spread. Uses HEOM in the presence of nominal factors; normalised Euclidean otherwise.
- mergen.metrics.cv_distances(X, space=None)[source]#
Coefficient of variation of pairwise distances: std / mean.
Lower values indicate more uniform spacing. Uses HEOM in the presence of nominal factors; normalised Euclidean otherwise.
References
- Lekivetz, R. & Jones, B. (2015).
Quality Engineering, 27(1), 46-52.
- mergen.metrics.minimax(X, space=None, n_eval=5000)[source]#
Minimax (fill / coverage) distance.
The maximum over evaluation grid points of the minimum distance to any design point. Lower values indicate better space coverage. Uses HEOM when space has nominal factors so that coverage is defined consistently on mixed spaces.
References
- Johnson, M. E., Moore, L. M. & Ylvisaker, D. (1990).
Journal of Statistical Planning and Inference, 26(2), 131-148.
- mergen.metrics.max_abs_correlation(X, space=None)[source]#
Maximum absolute pairwise correlation between parameter columns.
Lower values indicate better orthogonality between parameters. A value of 0 means parameters are completely uncorrelated.
Correlation is defined only on ordered columns, so nominal columns (when space is provided and lists any) are excluded from the pairing pool. If fewer than two ordered columns remain the metric returns 0.
- mergen.metrics.projection_cd2(X, space=None)[source]#
Mean centred L2 discrepancy across all 2D projections.
Evaluates uniformity in every pair of parameter dimensions. Lower values indicate better 2D projection coverage.
Nominal columns are excluded from the projection sweep because the CD2 kernel of Hickernell (1998) is defined on ordered \([0, 1]\) axes; averaging it over pairs that involve a nominal column would give a value with no interpretation.
References
- Liu, H. & Liu, M.-Q. (2023).
Canadian Journal of Statistics, 51(1), 293-311.
- Hickernell, F. J. (1998). Mathematics of Computation, 67(221),
299-322.
- mergen.metrics.quality_report(result, metrics='default', criteria_metrics=None, mc_samples=0, verbose=True)[source]#
Compute and print design quality metrics.
- Parameters:
result (SamplingResult)
metrics ('default' | list of metric names) – Which metrics to compute. Default: all 6 standard metrics.
criteria_metrics (list of criterion names or None) – Additional criteria to evaluate the design against (e.g. [‘maxpro’, ‘phi_p’]). Computed from the design directly, no SA involved. Useful to see how the design scores under criteria other than the one used for optimisation.
mc_samples (int) – Number of random designs drawn from the feasible space for baseline comparison. 0 = disabled (default). When > 0, each metric and criterion is given a percentile rank: what fraction of random designs does this design outperform?
verbose (bool)
- Returns:
dict with all computed values and (if mc_samples > 0) baseline stats.
- Return type:
- mergen.metrics.comparison(result)[source]#
Compare design quality metrics across all criteria in result.designs.
Each row is a criterion, each column is a metric. Only the standard metrics are included (no MC).
- Parameters:
result (SamplingResult)
- Returns:
pd.DataFrame — rows = criteria, columns = metrics
- Return type: