Note
Go to the end to download the full example code.
How large should a design be?#
Contrast a default-size and a doubled design of the same study to see what extra runs buy.
The same four-factor study is planned two ways: once under a hard experimental budget that fixes the number of runs, and once at the default size Mergen recommends. Placing the two designs side by side shows what a larger budget actually buys, so the decision can be made before any real experiment is spent.
The honest comparison here is coverage, not a quality percentile. Each design is optimised to fill the space as well as it can for its size, so both score well against a random baseline; a small design is not a “bad” design. What a small design cannot do is occupy as much of the space: with fewer points, larger gaps are unavoidable. That gap in coverage, visible in the pairplot, is the real cost of a tight budget.
Parameters#
factor_a, factor_b, factor_c, factor_d (0.0-1.0, continuous, 20-level grid, rounded to 3 decimals): four generic normalised inputs. With four factors the default sample size is 10*d = 40.
What to look at#
The two saved pairplots, compared directly: the 15-run design leaves visibly larger empty regions in several 2D projections, while the 40-run design places points into those gaps. This difference in coverage, not a difference in percentile score, is what a larger budget buys.
quality_report()for both runs (printed): note that both designs score well for their size. A high percentile confirms each design is well-spread relative to random designs of the same count; it does not mean 15 runs cover the space as fully as 40. Coverage and per-size quality are different questions, and the pairplots answer the coverage one.
Mergen features used#
Sampler.set_design(n_samples=...): an explicit fixed budget in the first run, versus omitting n_samples in the second so the 10*d default applies.Two independent designs from the same parameter space, compared on coverage (pairplots) and per-size quality (quality_report).
Sampler.set_optimizer(): a modest, shared compute budget so the two runs finish quickly for a demonstration.
Estimated runtime: a minute or two (two designs).
from mergen import ParameterSpace, Sampler
# 1. Define a four-factor numeric space. With d = 4 the default sample
# size is 10*d = 40; a rule of thumb from the computer-experiments
# literature that a design should scale with dimensionality.
space = ParameterSpace({
'factor_a': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}),
'factor_b': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}),
'factor_c': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}),
'factor_d': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}),
})
# 2. First study: a hard budget of only 15 runs.
budget_sampler = Sampler(space)
budget_sampler.set_design(n_samples=15)
budget_sampler.set_optimizer('sa', n_restarts=2, max_iter=300)
budget_design = budget_sampler.run()
# 3. Second study: let Mergen size the design (the 10*d default).
default_sampler = Sampler(space)
default_sampler.set_optimizer('sa', n_restarts=2, max_iter=300)
default_design = default_sampler.run()
# 4. Compare per-size quality (printed), then the coverage difference
# in the pairplots, which is the real cost of a smaller budget.
budget_design.quality_report()
default_design.quality_report()
budget_design.plot('pairplot', save=True)
default_design.plot('pairplot', save=True)
[WARNING] n_samples (15) < recommended 10*n_parameters (40, Loeppky et al. 2009). Design quality may be reduced.
════════════════════════════════════════════════════════════
MERGEN — Space-filling Design
════════════════════════════════════════════════════════════
Parameters : 4
Candidates : 160,000
n_samples : 15 (prescribed_in=0, focus_in=0, optimised_slots=15)
Total design : 15
Validation : 3
Criterion : umaxpro
Algorithm(s) : sa
────────────────────────────────────────────────────────────
[MERGEN] Optimising (criterion=umaxpro, algorithm=sa)...
[SA] Restart 1/2
[SA] Tuning temperature...
[SA] Start log(score)=30.177 T=1.3344e+23 iters=300 swappable=15 hybrid=0.50
[SA] Done log(score)=29.861 (accepted=280/300, rate=93.3%)
[SA] Restart 1: new best log(score)=29.861
[SA] Restart 2/2
[SA] Tuning temperature...
[SA] Start log(score)=50.050 T=2.8713e+28 iters=300 swappable=15 hybrid=0.50
[SA] Done log(score)=35.671 (accepted=277/300, rate=92.3%)
[MERGEN] sa done -- score=9.304e+12 (elapsed 4.6s)
────────────────────────────────────────────────────────────
MERGEN — Final Design
────────────────────────────────────────────────────────────
Prescribed (in) : 0
Prescribed (out) : 0
Focus (in) : 0
Focus (out) : 0
Optimised : 15
Total design : 15
Validation : 3
════════════════════════════════════════════════════════════
════════════════════════════════════════════════════════════
MERGEN — Space-filling Design
════════════════════════════════════════════════════════════
Parameters : 4
Candidates : 160,000
n_samples : 40 (prescribed_in=0, focus_in=0, optimised_slots=40)
Total design : 40
Validation : 8
Criterion : umaxpro
Algorithm(s) : sa
────────────────────────────────────────────────────────────
[MERGEN] Optimising (criterion=umaxpro, algorithm=sa)...
[SA] Restart 1/2
[SA] Tuning temperature...
[SA] Start log(score)=55.706 T=1.2958e+29 iters=300 swappable=40 hybrid=0.50
[SA] Done log(score)=53.240 (accepted=292/300, rate=97.3%)
[SA] Restart 1: new best log(score)=53.240
[SA] Restart 2/2
[SA] Tuning temperature...
[SA] Start log(score)=54.732 T=1.4286e+37 iters=300 swappable=40 hybrid=0.50
[SA] Done log(score)=54.417 (accepted=292/300, rate=97.3%)
[MERGEN] sa done -- score=1.323e+23 (elapsed 5.9s)
────────────────────────────────────────────────────────────
MERGEN — Final Design
────────────────────────────────────────────────────────────
Prescribed (in) : 0
Prescribed (out) : 0
Focus (in) : 0
Focus (out) : 0
Optimised : 40
Total design : 40
Validation : 8
════════════════════════════════════════════════════════════
[METRICS] Computing MC baseline (300 designs)...
[METRICS] MC baseline complete (300 designs).
════════════════════════════════════════════════════════════════════════
MERGEN Design Metrics (n=15, d=4)
════════════════════════════════════════════════════════════════════════
Metric Value Baseline Better when Rank
────────────────────────────────────────────────────────────────────────
Min distance 0.2683 0.2232 higher 78th pct
Minimax distance 0.8126 0.8773 lower 87th pct *
Max |correlation| 0.3977 0.4206 lower 59th pct *
2D projection CD2 0.0854 0.1584 lower 100th pct *
CV distances 0.3249 0.3090 lower 24th pct
Mean distance 0.8450 0.8211 higher 71th pct
────────────────────────────────────────────────────────────────────────
Criterion scores
────────────────────────────────────────────────────────────────────────
UMAXPRO 9.3041e+12 9.0264e+22 100th pct lower
────────────────────────────────────────────────────────────────────────
* = primarily optimised by 'umaxpro'
For other priorities, see: mergen.criteria.list_criteria()
────────────────────────────────────────────────────────────────────────
Baseline: 300 MC designs from feasible space | Rank = percentile among baseline designs
════════════════════════════════════════════════════════════════════════
[METRICS] Computing MC baseline (300 designs)...
[METRICS] MC baseline complete (300 designs).
════════════════════════════════════════════════════════════════════════
MERGEN Design Metrics (n=40, d=4)
════════════════════════════════════════════════════════════════════════
Metric Value Baseline Better when Rank
────────────────────────────────────────────────────────────────────────
Min distance 0.2168 0.1286 higher 100th pct
Minimax distance 0.6511 0.7075 lower 86th pct *
Max |correlation| 0.2342 0.2564 lower 62th pct *
2D projection CD2 0.0602 0.1015 lower 100th pct *
CV distances 0.3055 0.3169 lower 88th pct
Mean distance 0.8313 0.8183 higher 66th pct
────────────────────────────────────────────────────────────────────────
Criterion scores
────────────────────────────────────────────────────────────────────────
UMAXPRO 1.3233e+23 1.7280e+25 100th pct lower
────────────────────────────────────────────────────────────────────────
* = primarily optimised by 'umaxpro'
For other priorities, see: mergen.criteria.list_criteria()
────────────────────────────────────────────────────────────────────────
Baseline: 300 MC designs from feasible space | Rank = percentile among baseline designs
════════════════════════════════════════════════════════════════════════
Saved: outputs/pairplot_5.png
Saved: outputs/pairplot_6.png
Total running time of the script: (0 minutes 35.365 seconds)