.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/13_ml_hyperparameters.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_13_ml_hyperparameters.py: Machine-learning hyperparameters ================================ Cover a hyperparameter space with far fewer runs than the full grid would need. A hyperparameter design is built for training a model over four knobs: learning rate, batch size, optimiser, and weight decay. Instead of a full grid (which explodes combinatorially) or random search (which clusters and leaves gaps), a space-filling design covers the configuration space evenly with a modest number of runs. The optimiser is a nominal factor, so the design is scored with maxproqq. A fixed baseline configuration the team always wants to include is attached as its own set. Parameters ---------- - learning_rate (discrete decades: 1e-4, 3e-4, 1e-3, 3e-3, 1e-2): sampled on a log-like ladder rather than linearly, as is standard for learning rates. - batch_size (discrete powers of two: 16, 32, 64, 128, 256): the usual hardware-friendly choices. - optimiser (nominal: 'adam', 'sgd', 'rmsprop'): unordered categorical. - weight_decay (discrete: 0.0, 1e-4, 1e-3, 1e-2): common regularisation strengths. What to look at --------------- - ``summary()``: the space-filling configurations plus the always-included baseline set; note how few runs cover the space compared with a full grid (5 x 5 x 3 x 4 = 300 combinations). - The saved pairplot: even coverage across the numeric knobs, with all three optimisers visited; the baseline configuration appears in its own colour. - ``configs.json``: the design as JSON, the natural format when each row is a configuration consumed directly by a training script. Mergen features used -------------------- - Log-like discrete numeric factors alongside a nominal factor. - ``criteria='maxproqq'`` as the correct choice for the nominal optimiser. - ``Sampler.add_set()``: pin a fixed baseline configuration that must always be part of the design. - to_json export for configurations consumed by code. Estimated runtime: a few seconds to a minute. .. GENERATED FROM PYTHON SOURCE LINES 48-72 .. code-block:: Python from mergen import ParameterSpace, Sampler # 1. Define the hyperparameter space. space = ParameterSpace({ 'learning_rate': [1e-4, 3e-4, 1e-3, 3e-3, 1e-2], 'batch_size': [16, 32, 64, 128, 256], 'optimiser': ('nominal', ['adam', 'sgd', 'rmsprop']), 'weight_decay': [0.0, 1e-4, 1e-3, 1e-2], }) # 2. Always include a known-good baseline configuration. sampler = Sampler(space) sampler.add_set('baseline', [[1e-3, 32, 'adam', 1e-4]], color='#ff8800') # 3. Build the space-filling design with maxproqq (nominal optimiser). sampler.set_design(n_samples=20, n_validation=4) result = sampler.run(criteria='maxproqq') # 4. Inspect and export the configurations for the training script. result.summary() result.plot('pairplot', save=True) result.to_json('configs.json') .. rst-class:: sphx-glr-script-out .. code-block:: none [WARNING] n_samples (20) < recommended 10*n_parameters (40, Loeppky et al. 2009). Design quality may be reduced. ════════════════════════════════════════════════════════════ MERGEN — Space-filling Design ════════════════════════════════════════════════════════════ Parameters : 4 Candidates : 300 n_samples : 20 (prescribed_in=0, focus_in=0, optimised_slots=20) Total design : 20 Validation : 4 Criterion : maxproqq Algorithm(s) : sa ──────────────────────────────────────────────────────────── [MERGEN] Optimising (criterion=maxproqq, algorithm=sa)... [SA] Restart 1/5 [SA] Tuning temperature... [SA] Start log(score)=12.725 T=1.7640e+05 iters=2000 swappable=20 hybrid=0.50 iter 500/2000 T=1.764e+04 best log(score)=12.075 iter 1000/2000 T=1.764e+03 best log(score)=12.075 iter 1500/2000 T=1.764e+02 best log(score)=12.024 iter 2000/2000 T=1.764e+01 best log(score)=12.024 [SA] Done log(score)=12.024 (accepted=1678/2000, rate=83.9%) [SA] Restart 1: new best log(score)=12.024 [SA] Restart 2/5 [SA] Tuning temperature... [SA] Start log(score)=12.385 T=1.6610e+05 iters=2000 swappable=20 hybrid=0.50 iter 500/2000 T=1.661e+04 best log(score)=11.856 iter 1000/2000 T=1.661e+03 best log(score)=11.856 iter 1500/2000 T=1.661e+02 best log(score)=11.856 iter 2000/2000 T=1.661e+01 best log(score)=11.856 [SA] Done log(score)=11.856 (accepted=1639/2000, rate=82.0%) [SA] Restart 2: new best log(score)=11.856 [SA] Restart 3/5 [SA] Tuning temperature... [SA] Start log(score)=12.355 T=1.9463e+05 iters=2000 swappable=20 hybrid=0.50 iter 500/2000 T=1.946e+04 best log(score)=11.881 iter 1000/2000 T=1.946e+03 best log(score)=11.881 iter 1500/2000 T=1.946e+02 best log(score)=11.861 iter 2000/2000 T=1.946e+01 best log(score)=11.861 [SA] Done log(score)=11.861 (accepted=1665/2000, rate=83.2%) [SA] Restart 4/5 [SA] Tuning temperature... [SA] Start log(score)=12.191 T=1.4104e+05 iters=2000 swappable=20 hybrid=0.50 iter 500/2000 T=1.410e+04 best log(score)=11.884 iter 1000/2000 T=1.410e+03 best log(score)=11.884 iter 1500/2000 T=1.410e+02 best log(score)=11.884 iter 2000/2000 T=1.410e+01 best log(score)=11.884 [SA] Done log(score)=11.884 (accepted=1641/2000, rate=82.0%) [SA] Restart 5/5 [SA] Tuning temperature... [SA] Start log(score)=12.911 T=1.4254e+05 iters=2000 swappable=20 hybrid=0.50 iter 500/2000 T=1.425e+04 best log(score)=11.928 iter 1000/2000 T=1.425e+03 best log(score)=11.896 iter 1500/2000 T=1.425e+02 best log(score)=11.896 iter 2000/2000 T=1.425e+01 best log(score)=11.784 [SA] Done log(score)=11.784 (accepted=1624/2000, rate=81.2%) [SA] Restart 5: new best log(score)=11.784 [MERGEN] sa done -- score=1.311e+05 (elapsed 10.0s) ──────────────────────────────────────────────────────────── MERGEN — Final Design ──────────────────────────────────────────────────────────── Prescribed (in) : 0 Prescribed (out) : 0 Focus (in) : 0 Focus (out) : 0 Optimised : 20 Total design : 20 Validation : 4 baseline : 1 ════════════════════════════════════════════════════════════ ──────────────────────────────────────────────────── MERGEN Design Summary ──────────────────────────────────────────────────── Optimised : 20 Total design : 20 Validation : 4 baseline : 1 ──────────────────────────────────────────────────── Parameters : 4 Candidates : 300 Criterion : maxproqq Seed : 44 Algorithm : sa ──────────────────────────────────────────────────── Saved: outputs/pairplot_15.png Saved: outputs/configs.json (25 rows) .. image-sg:: /auto_examples/images/sphx_glr_13_ml_hyperparameters_001.png :alt: 13 ml hyperparameters :srcset: /auto_examples/images/sphx_glr_13_ml_hyperparameters_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 48-48 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 10.888 seconds) .. _sphx_glr_download_auto_examples_13_ml_hyperparameters.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 13_ml_hyperparameters.ipynb <13_ml_hyperparameters.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 13_ml_hyperparameters.py <13_ml_hyperparameters.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 13_ml_hyperparameters.zip <13_ml_hyperparameters.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_