Note
Go to the end to download the full example code.
Extra sets: adding a test set#
Add a hand-picked test set alongside the optimised design and the automatic validation split.
A machine-learning surrogate workflow needs three disjoint sets of points drawn from one input space: a space-filling training design, an automatic validation set, and an external held-out test set whose points the team has chosen by hand. This example shows how Mergen keeps all three separate, with no point shared between them.
Parameters#
x1, x2, x3 (discrete grids): three inputs defined on explicit grids so the hand-picked test points land exactly on grid nodes. x1 and x2 are integer ladders; x3 is a stepped list.
What to look at#
summary(): three distinct groups are reported, ‘Optimised’ (the training design), ‘Validation’ (the automatic hold-out) and ‘test’ (the user set); their counts should not overlap.The saved pairplot: the training points, validation points and the user ‘test’ points appear in three different colours, with the test points sitting exactly where they were prescribed and never reused by the training design.
design.csv: all three groups in one file, distinguished by the point_type column, ready to feed a training / validation / test split downstream.
Mergen features used#
Sampler.add_set('test', [...], color=...): attach an external, user-chosen point set with its own colour; its nodes are reserved so the optimiser cannot select them.Sampler.set_design(n_validation=...): request the automatic validation hold-out alongside the training design.
Estimated runtime: a few seconds to a minute.
from mergen import ParameterSpace, Sampler
# 1. Define the input space on discrete grids so the hand-picked test
# points are exact grid nodes.
space = ParameterSpace({
'x1': range(0, 101, 10), # 0, 10, ..., 100
'x2': range(0, 101, 10),
'x3': [0.0, 0.25, 0.5, 0.75, 1.0],
})
# 2. Attach a hand-picked external test set in its own colour.
sampler = Sampler(space)
sampler.add_set('test',
[[0, 100, 0.0], [100, 0, 1.0], [50, 50, 0.5]],
color='#9b5de5')
# 3. Ask for an automatic validation hold-out alongside the training
# design, then build all groups in one run.
sampler.set_design(n_samples=25, n_validation=5)
result = sampler.run()
# 4. Confirm the three groups are separate, then save.
result.summary()
result.plot('pairplot', save=True)
result.to_csv('design.csv')
[WARNING] n_samples (25) < recommended 10*n_parameters (30, Loeppky et al. 2009). Design quality may be reduced.
════════════════════════════════════════════════════════════
MERGEN — Space-filling Design
════════════════════════════════════════════════════════════
Parameters : 3
Candidates : 605
n_samples : 25 (prescribed_in=0, focus_in=0, optimised_slots=25)
Total design : 25
Validation : 5
Criterion : umaxpro
Algorithm(s) : sa
────────────────────────────────────────────────────────────
[MERGEN] Optimising (criterion=umaxpro, algorithm=sa)...
[SA] Restart 1/5
[SA] Tuning temperature...
[SA] Start log(score)=51.603 T=1.4286e+27 iters=2500 swappable=25 hybrid=0.50
iter 500/2500 T=2.264e+26 best log(score)=51.599
iter 1000/2500 T=3.588e+25 best log(score)=51.069
iter 1500/2500 T=5.687e+24 best log(score)=50.983
iter 2000/2500 T=9.014e+23 best log(score)=50.983
iter 2500/2500 T=1.429e+23 best log(score)=50.983
[SA] Done log(score)=50.983 (accepted=2253/2500, rate=90.1%)
[SA] Restart 1: new best log(score)=50.983
[SA] Restart 2/5
[SA] Tuning temperature...
[SA] Start log(score)=69.078 T=1.4286e+27 iters=2500 swappable=25 hybrid=0.50
iter 500/2500 T=2.264e+26 best log(score)=50.949
iter 1000/2500 T=3.588e+25 best log(score)=50.949
iter 1500/2500 T=5.687e+24 best log(score)=50.949
iter 2000/2500 T=9.014e+23 best log(score)=50.609
iter 2500/2500 T=1.429e+23 best log(score)=50.609
[SA] Done log(score)=50.609 (accepted=2224/2500, rate=89.0%)
[SA] Restart 2: new best log(score)=50.609
[SA] Restart 3/5
[SA] Tuning temperature...
[SA] Start log(score)=52.009 T=4.2857e+27 iters=2500 swappable=25 hybrid=0.50
iter 500/2500 T=6.792e+26 best log(score)=51.178
iter 1000/2500 T=1.077e+26 best log(score)=51.178
iter 1500/2500 T=1.706e+25 best log(score)=50.109
iter 2000/2500 T=2.704e+24 best log(score)=50.109
iter 2500/2500 T=4.286e+23 best log(score)=50.109
[SA] Done log(score)=50.109 (accepted=2254/2500, rate=90.2%)
[SA] Restart 3: new best log(score)=50.109
[SA] Restart 4/5
[SA] Tuning temperature...
[SA] Start log(score)=51.494 T=1.4286e+27 iters=2500 swappable=25 hybrid=0.50
iter 500/2500 T=2.264e+26 best log(score)=50.668
iter 1000/2500 T=3.588e+25 best log(score)=50.668
iter 1500/2500 T=5.687e+24 best log(score)=50.668
iter 2000/2500 T=9.014e+23 best log(score)=50.668
iter 2500/2500 T=1.429e+23 best log(score)=50.668
[SA] Done log(score)=50.668 (accepted=2255/2500, rate=90.2%)
[SA] Restart 5/5
[SA] Tuning temperature...
[SA] Start log(score)=51.369 T=2.8571e+27 iters=2500 swappable=25 hybrid=0.50
iter 500/2500 T=4.528e+26 best log(score)=50.681
iter 1000/2500 T=7.177e+25 best log(score)=50.332
iter 1500/2500 T=1.137e+25 best log(score)=50.332
iter 2000/2500 T=1.803e+24 best log(score)=50.332
iter 2500/2500 T=2.857e+23 best log(score)=50.332
[SA] Done log(score)=50.332 (accepted=2258/2500, rate=90.3%)
[MERGEN] sa done -- score=5.783e+21 (elapsed 8.8s)
────────────────────────────────────────────────────────────
MERGEN — Final Design
────────────────────────────────────────────────────────────
Prescribed (in) : 0
Prescribed (out) : 0
Focus (in) : 0
Focus (out) : 0
Optimised : 25
Total design : 25
Validation : 5
test : 3
════════════════════════════════════════════════════════════
────────────────────────────────────────────────────
MERGEN Design Summary
────────────────────────────────────────────────────
Optimised : 25
Total design : 25
Validation : 5
test : 3
────────────────────────────────────────────────────
Parameters : 3
Candidates : 605
Criterion : umaxpro
Seed : 44
Algorithm : sa
────────────────────────────────────────────────────
Saved: outputs/pairplot_7.png
Saved: outputs/design.csv (33 rows)
Total running time of the script: (0 minutes 9.306 seconds)