Note
Go to the end to download the full example code.
Machine-learning hyperparameters#
Cover a hyperparameter space with far fewer runs than the full grid would need.
A hyperparameter design is built for training a model over four knobs: learning rate, batch size, optimiser, and weight decay. Instead of a full grid (which explodes combinatorially) or random search (which clusters and leaves gaps), a space-filling design covers the configuration space evenly with a modest number of runs. The optimiser is a nominal factor, so the design is scored with maxproqq. A fixed baseline configuration the team always wants to include is attached as its own set.
Parameters#
learning_rate (discrete decades: 1e-4, 3e-4, 1e-3, 3e-3, 1e-2): sampled on a log-like ladder rather than linearly, as is standard for learning rates.
batch_size (discrete powers of two: 16, 32, 64, 128, 256): the usual hardware-friendly choices.
optimiser (nominal: ‘adam’, ‘sgd’, ‘rmsprop’): unordered categorical.
weight_decay (discrete: 0.0, 1e-4, 1e-3, 1e-2): common regularisation strengths.
What to look at#
summary(): the space-filling configurations plus the always-included baseline set; note how few runs cover the space compared with a full grid (5 x 5 x 3 x 4 = 300 combinations).The saved pairplot: even coverage across the numeric knobs, with all three optimisers visited; the baseline configuration appears in its own colour.
configs.json: the design as JSON, the natural format when each row is a configuration consumed directly by a training script.
Mergen features used#
Log-like discrete numeric factors alongside a nominal factor.
criteria='maxproqq'as the correct choice for the nominal optimiser.Sampler.add_set(): pin a fixed baseline configuration that must always be part of the design.to_json export for configurations consumed by code.
Estimated runtime: a few seconds to a minute.
from mergen import ParameterSpace, Sampler
# 1. Define the hyperparameter space.
space = ParameterSpace({
'learning_rate': [1e-4, 3e-4, 1e-3, 3e-3, 1e-2],
'batch_size': [16, 32, 64, 128, 256],
'optimiser': ('nominal', ['adam', 'sgd', 'rmsprop']),
'weight_decay': [0.0, 1e-4, 1e-3, 1e-2],
})
# 2. Always include a known-good baseline configuration.
sampler = Sampler(space)
sampler.add_set('baseline',
[[1e-3, 32, 'adam', 1e-4]],
color='#ff8800')
# 3. Build the space-filling design with maxproqq (nominal optimiser).
sampler.set_design(n_samples=20, n_validation=4)
result = sampler.run(criteria='maxproqq')
# 4. Inspect and export the configurations for the training script.
result.summary()
result.plot('pairplot', save=True)
result.to_json('configs.json')
[WARNING] n_samples (20) < recommended 10*n_parameters (40, Loeppky et al. 2009). Design quality may be reduced.
════════════════════════════════════════════════════════════
MERGEN — Space-filling Design
════════════════════════════════════════════════════════════
Parameters : 4
Candidates : 300
n_samples : 20 (prescribed_in=0, focus_in=0, optimised_slots=20)
Total design : 20
Validation : 4
Criterion : maxproqq
Algorithm(s) : sa
────────────────────────────────────────────────────────────
[MERGEN] Optimising (criterion=maxproqq, algorithm=sa)...
[SA] Restart 1/5
[SA] Tuning temperature...
[SA] Start log(score)=12.725 T=1.7640e+05 iters=2000 swappable=20 hybrid=0.50
iter 500/2000 T=1.764e+04 best log(score)=12.075
iter 1000/2000 T=1.764e+03 best log(score)=12.075
iter 1500/2000 T=1.764e+02 best log(score)=12.024
iter 2000/2000 T=1.764e+01 best log(score)=12.024
[SA] Done log(score)=12.024 (accepted=1678/2000, rate=83.9%)
[SA] Restart 1: new best log(score)=12.024
[SA] Restart 2/5
[SA] Tuning temperature...
[SA] Start log(score)=12.385 T=1.6610e+05 iters=2000 swappable=20 hybrid=0.50
iter 500/2000 T=1.661e+04 best log(score)=11.856
iter 1000/2000 T=1.661e+03 best log(score)=11.856
iter 1500/2000 T=1.661e+02 best log(score)=11.856
iter 2000/2000 T=1.661e+01 best log(score)=11.856
[SA] Done log(score)=11.856 (accepted=1639/2000, rate=82.0%)
[SA] Restart 2: new best log(score)=11.856
[SA] Restart 3/5
[SA] Tuning temperature...
[SA] Start log(score)=12.355 T=1.9463e+05 iters=2000 swappable=20 hybrid=0.50
iter 500/2000 T=1.946e+04 best log(score)=11.881
iter 1000/2000 T=1.946e+03 best log(score)=11.881
iter 1500/2000 T=1.946e+02 best log(score)=11.861
iter 2000/2000 T=1.946e+01 best log(score)=11.861
[SA] Done log(score)=11.861 (accepted=1665/2000, rate=83.2%)
[SA] Restart 4/5
[SA] Tuning temperature...
[SA] Start log(score)=12.191 T=1.4104e+05 iters=2000 swappable=20 hybrid=0.50
iter 500/2000 T=1.410e+04 best log(score)=11.884
iter 1000/2000 T=1.410e+03 best log(score)=11.884
iter 1500/2000 T=1.410e+02 best log(score)=11.884
iter 2000/2000 T=1.410e+01 best log(score)=11.884
[SA] Done log(score)=11.884 (accepted=1641/2000, rate=82.0%)
[SA] Restart 5/5
[SA] Tuning temperature...
[SA] Start log(score)=12.911 T=1.4254e+05 iters=2000 swappable=20 hybrid=0.50
iter 500/2000 T=1.425e+04 best log(score)=11.928
iter 1000/2000 T=1.425e+03 best log(score)=11.896
iter 1500/2000 T=1.425e+02 best log(score)=11.896
iter 2000/2000 T=1.425e+01 best log(score)=11.784
[SA] Done log(score)=11.784 (accepted=1624/2000, rate=81.2%)
[SA] Restart 5: new best log(score)=11.784
[MERGEN] sa done -- score=1.311e+05 (elapsed 10.0s)
────────────────────────────────────────────────────────────
MERGEN — Final Design
────────────────────────────────────────────────────────────
Prescribed (in) : 0
Prescribed (out) : 0
Focus (in) : 0
Focus (out) : 0
Optimised : 20
Total design : 20
Validation : 4
baseline : 1
════════════════════════════════════════════════════════════
────────────────────────────────────────────────────
MERGEN Design Summary
────────────────────────────────────────────────────
Optimised : 20
Total design : 20
Validation : 4
baseline : 1
────────────────────────────────────────────────────
Parameters : 4
Candidates : 300
Criterion : maxproqq
Seed : 44
Algorithm : sa
────────────────────────────────────────────────────
Saved: outputs/pairplot_15.png
Saved: outputs/configs.json (25 rows)
Total running time of the script: (0 minutes 10.888 seconds)