.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/06_sample_size.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_06_sample_size.py: How large should a design be? ============================= Contrast a default-size and a doubled design of the same study to see what extra runs buy. The same four-factor study is planned two ways: once under a hard experimental budget that fixes the number of runs, and once at the default size Mergen recommends. Placing the two designs side by side shows what a larger budget actually buys, so the decision can be made before any real experiment is spent. The honest comparison here is coverage, not a quality percentile. Each design is optimised to fill the space as well as it can for its size, so both score well against a random baseline; a small design is not a "bad" design. What a small design cannot do is occupy as much of the space: with fewer points, larger gaps are unavoidable. That gap in coverage, visible in the pairplot, is the real cost of a tight budget. Parameters ---------- - factor_a, factor_b, factor_c, factor_d (0.0-1.0, continuous, 20-level grid, rounded to 3 decimals): four generic normalised inputs. With four factors the default sample size is 10*d = 40. What to look at --------------- - The two saved pairplots, compared directly: the 15-run design leaves visibly larger empty regions in several 2D projections, while the 40-run design places points into those gaps. This difference in coverage, not a difference in percentile score, is what a larger budget buys. - ``quality_report()`` for both runs (printed): note that both designs score well for their size. A high percentile confirms each design is well-spread relative to random designs of the same count; it does not mean 15 runs cover the space as fully as 40. Coverage and per-size quality are different questions, and the pairplots answer the coverage one. Mergen features used -------------------- - ``Sampler.set_design(n_samples=...)``: an explicit fixed budget in the first run, versus omitting n_samples in the second so the 10*d default applies. - Two independent designs from the same parameter space, compared on coverage (pairplots) and per-size quality (quality_report). - ``Sampler.set_optimizer()``: a modest, shared compute budget so the two runs finish quickly for a demonstration. Estimated runtime: a minute or two (two designs). .. GENERATED FROM PYTHON SOURCE LINES 52-81 .. code-block:: Python from mergen import ParameterSpace, Sampler # 1. Define a four-factor numeric space. With d = 4 the default sample # size is 10*d = 40; a rule of thumb from the computer-experiments # literature that a design should scale with dimensionality. space = ParameterSpace({ 'factor_a': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}), 'factor_b': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}), 'factor_c': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}), 'factor_d': ('continuous', 0.0, 1.0, {'resolution': 20, 'round': 3}), }) # 2. First study: a hard budget of only 15 runs. budget_sampler = Sampler(space) budget_sampler.set_design(n_samples=15) budget_sampler.set_optimizer('sa', n_restarts=2, max_iter=300) budget_design = budget_sampler.run() # 3. Second study: let Mergen size the design (the 10*d default). default_sampler = Sampler(space) default_sampler.set_optimizer('sa', n_restarts=2, max_iter=300) default_design = default_sampler.run() # 4. Compare per-size quality (printed), then the coverage difference # in the pairplots, which is the real cost of a smaller budget. budget_design.quality_report() default_design.quality_report() budget_design.plot('pairplot', save=True) default_design.plot('pairplot', save=True) .. rst-class:: sphx-glr-script-out .. code-block:: none [WARNING] n_samples (15) < recommended 10*n_parameters (40, Loeppky et al. 2009). Design quality may be reduced. ════════════════════════════════════════════════════════════ MERGEN — Space-filling Design ════════════════════════════════════════════════════════════ Parameters : 4 Candidates : 160,000 n_samples : 15 (prescribed_in=0, focus_in=0, optimised_slots=15) Total design : 15 Validation : 3 Criterion : umaxpro Algorithm(s) : sa ──────────────────────────────────────────────────────────── [MERGEN] Optimising (criterion=umaxpro, algorithm=sa)... [SA] Restart 1/2 [SA] Tuning temperature... [SA] Start log(score)=30.177 T=1.3344e+23 iters=300 swappable=15 hybrid=0.50 [SA] Done log(score)=29.861 (accepted=280/300, rate=93.3%) [SA] Restart 1: new best log(score)=29.861 [SA] Restart 2/2 [SA] Tuning temperature... [SA] Start log(score)=50.050 T=2.8713e+28 iters=300 swappable=15 hybrid=0.50 [SA] Done log(score)=35.671 (accepted=277/300, rate=92.3%) [MERGEN] sa done -- score=9.304e+12 (elapsed 3.3s) ──────────────────────────────────────────────────────────── MERGEN — Final Design ──────────────────────────────────────────────────────────── Prescribed (in) : 0 Prescribed (out) : 0 Focus (in) : 0 Focus (out) : 0 Optimised : 15 Total design : 15 Validation : 3 ════════════════════════════════════════════════════════════ ════════════════════════════════════════════════════════════ MERGEN — Space-filling Design ════════════════════════════════════════════════════════════ Parameters : 4 Candidates : 160,000 n_samples : 40 (prescribed_in=0, focus_in=0, optimised_slots=40) Total design : 40 Validation : 8 Criterion : umaxpro Algorithm(s) : sa ──────────────────────────────────────────────────────────── [MERGEN] Optimising (criterion=umaxpro, algorithm=sa)... [SA] Restart 1/2 [SA] Tuning temperature... [SA] Start log(score)=55.706 T=1.2958e+29 iters=300 swappable=40 hybrid=0.50 [SA] Done log(score)=53.240 (accepted=292/300, rate=97.3%) [SA] Restart 1: new best log(score)=53.240 [SA] Restart 2/2 [SA] Tuning temperature... [SA] Start log(score)=54.732 T=1.4286e+37 iters=300 swappable=40 hybrid=0.50 [SA] Done log(score)=54.417 (accepted=292/300, rate=97.3%) [MERGEN] sa done -- score=1.323e+23 (elapsed 4.2s) ──────────────────────────────────────────────────────────── MERGEN — Final Design ──────────────────────────────────────────────────────────── Prescribed (in) : 0 Prescribed (out) : 0 Focus (in) : 0 Focus (out) : 0 Optimised : 40 Total design : 40 Validation : 8 ════════════════════════════════════════════════════════════ [METRICS] Computing MC baseline (300 designs)... [METRICS] MC baseline complete (300 designs). ════════════════════════════════════════════════════════════════════════ MERGEN Design Metrics (n=15, d=4) ════════════════════════════════════════════════════════════════════════ Metric Value Baseline Better when Rank ──────────────────────────────────────────────────────────────────────── Min distance 0.2683 0.2232 higher 78th pct Minimax distance 0.8126 0.8773 lower 87th pct * Max |correlation| 0.3977 0.4206 lower 59th pct * 2D projection CD2 0.0854 0.1584 lower 100th pct * CV distances 0.3249 0.3090 lower 24th pct Mean distance 0.8450 0.8211 higher 71th pct ──────────────────────────────────────────────────────────────────────── Criterion scores ──────────────────────────────────────────────────────────────────────── UMAXPRO 9.3041e+12 9.0264e+22 100th pct lower ──────────────────────────────────────────────────────────────────────── * = primarily optimised by 'umaxpro' For other priorities, see: mergen.criteria.list_criteria() ──────────────────────────────────────────────────────────────────────── Baseline: 300 MC designs from feasible space | Rank = percentile among baseline designs ════════════════════════════════════════════════════════════════════════ [METRICS] Computing MC baseline (300 designs)... [METRICS] MC baseline complete (300 designs). ════════════════════════════════════════════════════════════════════════ MERGEN Design Metrics (n=40, d=4) ════════════════════════════════════════════════════════════════════════ Metric Value Baseline Better when Rank ──────────────────────────────────────────────────────────────────────── Min distance 0.2168 0.1286 higher 100th pct Minimax distance 0.6511 0.7075 lower 86th pct * Max |correlation| 0.2342 0.2564 lower 62th pct * 2D projection CD2 0.0602 0.1015 lower 100th pct * CV distances 0.3055 0.3169 lower 88th pct Mean distance 0.8313 0.8183 higher 66th pct ──────────────────────────────────────────────────────────────────────── Criterion scores ──────────────────────────────────────────────────────────────────────── UMAXPRO 1.3233e+23 1.7280e+25 100th pct lower ──────────────────────────────────────────────────────────────────────── * = primarily optimised by 'umaxpro' For other priorities, see: mergen.criteria.list_criteria() ──────────────────────────────────────────────────────────────────────── Baseline: 300 MC designs from feasible space | Rank = percentile among baseline designs ════════════════════════════════════════════════════════════════════════ Saved: outputs/pairplot_5.png Saved: outputs/pairplot_6.png .. rst-class:: sphx-glr-horizontal * .. image-sg:: /auto_examples/images/sphx_glr_06_sample_size_001.png :alt: 06 sample size :srcset: /auto_examples/images/sphx_glr_06_sample_size_001.png :class: sphx-glr-multi-img * .. image-sg:: /auto_examples/images/sphx_glr_06_sample_size_002.png :alt: 06 sample size :srcset: /auto_examples/images/sphx_glr_06_sample_size_002.png :class: sphx-glr-multi-img .. GENERATED FROM PYTHON SOURCE LINES 52-52 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 27.705 seconds) .. _sphx_glr_download_auto_examples_06_sample_size.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 06_sample_size.ipynb <06_sample_size.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 06_sample_size.py <06_sample_size.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 06_sample_size.zip <06_sample_size.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_