benjamc: Add test for Hyperband calculation

development/_modules/smac/intensifier/hyperband_utils.html

@@ -1075,7 +1075,7 @@ <h1>Source code for smac.intensifier.hyperband_utils</h1><div class="highlight">
         <span class="s2">&quot;budgets_in_stage&quot;</span><span class="p">:</span> <span class="n">_budgets_in_stage</span><span class="p">,</span>
         <span class="s2">&quot;trials_used&quot;</span><span class="p">:</span> <span class="n">total_trials</span><span class="p">,</span>
         <span class="s2">&quot;budget_used&quot;</span><span class="p">:</span> <span class="n">total_budget</span><span class="p">,</span>
-        <span class="s2">&quot;number_of_brackets&quot;</span><span class="p">:</span> <span class="n">_s_max</span><span class="p">,</span>
+        <span class="s2">&quot;number_of_brackets&quot;</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">_max_iterations</span><span class="p">),</span>
     <span class="p">}</span></div>
 
 
@@ -1101,13 +1101,19 @@ <h1>Source code for smac.intensifier.hyperband_utils</h1><div class="highlight">
 <span class="sd">    Returns</span>
 <span class="sd">    -------</span>
 <span class="sd">    dict</span>
-<span class="sd">        Info about the Hyperband round</span>
+<span class="sd">        Info about one Hyperband round</span>
 <span class="sd">            &quot;max_iterations&quot;</span>
 <span class="sd">            &quot;n_configs_in_stage&quot;</span>
 <span class="sd">            &quot;budgets_in_stage&quot;</span>
 <span class="sd">            &quot;trials_used&quot;</span>
 <span class="sd">            &quot;budget_used&quot;</span>
 <span class="sd">            &quot;number_of_brackets&quot;</span>
+<span class="sd">        Info about whole optimization</span>
+<span class="sd">            &quot;n_trials&quot;</span>
+<span class="sd">            &quot;total_budget&quot;</span>
+<span class="sd">            &quot;eta&quot;</span>
+<span class="sd">            &quot;min_budget&quot;</span>
+<span class="sd">            &quot;max_budget&quot;</span>
 
 <span class="sd">    &quot;&quot;&quot;</span>
     <span class="c1"># Determine the HB</span>
@@ -1135,9 +1141,6 @@ <h1>Source code for smac.intensifier.hyperband_utils</h1><div class="highlight">
             <span class="c1"># budget left. We can not add full c but the number of trials that still fit</span>
             <span class="n">remaining_budget</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">remaining_budget</span> <span class="o">-</span> <span class="n">b</span> <span class="o">*</span> <span class="n">c</span><span class="p">)</span>
 
-            <span class="c1"># print(stage, b, c)</span>
-            <span class="c1"># print(&quot;-&quot;*20, remaining_trials, remaining_budget)</span>
-
     <span class="n">n_trials</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">number_of_full_hb_rounds</span> <span class="o">*</span> <span class="n">trials_used_per_hb_round</span> <span class="o">+</span> <span class="n">remaining_trials</span><span class="p">)</span>
 
     <span class="n">hyperband_info</span> <span class="o">=</span> <span class="n">hyperband_round</span>

development/_sources/examples/1_basics/1_quadratic_function.rst.txt

@@ -161,7 +161,7 @@ be applied to problems with large evaluation budgets (up to 1000 evaluations).
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 4.231 seconds)
+   **Total running time of the script:** (0 minutes 4.219 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_1_quadratic_function.py:

development/_sources/examples/1_basics/2_svm_cv.rst.txt

@@ -387,7 +387,7 @@ types as well as conditional hyperparameters.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 2.887 seconds)
+   **Total running time of the script:** (0 minutes 2.883 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_2_svm_cv.py:

development/_sources/examples/1_basics/3_ask_and_tell.rst.txt

@@ -136,7 +136,7 @@ This examples show how to use the Ask-and-Tell interface.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 3.863 seconds)
+   **Total running time of the script:** (0 minutes 3.825 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_3_ask_and_tell.py:

development/_sources/examples/1_basics/4_callback.rst.txt

@@ -146,7 +146,7 @@ Furthermore, we print some stages of the optimization process.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 0.160 seconds)
+   **Total running time of the script:** (0 minutes 0.162 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_4_callback.py:

development/_sources/examples/1_basics/5_continue.rst.txt

@@ -172,7 +172,7 @@ there already is a previous run with the same meta data, this run will be contin
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 1.496 seconds)
+   **Total running time of the script:** (0 minutes 1.494 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_5_continue.py:

development/_sources/examples/1_basics/6_priors.rst.txt

@@ -229,7 +229,7 @@ optimization, you have to change the acquisition function to ``PriorAcquisitionF
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 39.626 seconds)
+   **Total running time of the script:** (0 minutes 39.275 seconds)
 
 
 .. _sphx_glr_download_examples_1_basics_6_priors.py:

development/_sources/examples/1_basics/sg_execution_times.rst.txt

@@ -6,7 +6,7 @@
 
 Computation times
 =================
-**00:52.264** total execution time for 6 files **from examples/1_basics**:
+**00:51.857** total execution time for 6 files **from examples/1_basics**:
 
 .. container::
 
@@ -33,20 +33,20 @@ Computation times
      - Time
      - Mem (MB)
    * - :ref:`sphx_glr_examples_1_basics_6_priors.py` (``6_priors.py``)
-     - 00:39.626
+     - 00:39.275
      - 0.0
    * - :ref:`sphx_glr_examples_1_basics_1_quadratic_function.py` (``1_quadratic_function.py``)
-     - 00:04.231
+     - 00:04.219
      - 0.0
    * - :ref:`sphx_glr_examples_1_basics_3_ask_and_tell.py` (``3_ask_and_tell.py``)
-     - 00:03.863
+     - 00:03.825
      - 0.0
    * - :ref:`sphx_glr_examples_1_basics_2_svm_cv.py` (``2_svm_cv.py``)
-     - 00:02.887
+     - 00:02.883
      - 0.0
    * - :ref:`sphx_glr_examples_1_basics_5_continue.py` (``5_continue.py``)
-     - 00:01.496
+     - 00:01.494
      - 0.0
    * - :ref:`sphx_glr_examples_1_basics_4_callback.py` (``4_callback.py``)
-     - 00:00.160
+     - 00:00.162
      - 0.0

development/_sources/examples/2_multi_fidelity/1_mlp_epochs.rst.txt

@@ -68,8 +68,14 @@ is chosen to optimize the average accuracy on 5-fold cross validation.
     [INFO][smbo.py:320] Finished 0 trials.
     [INFO][smbo.py:320] Finished 100 trials.
     [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
+    [INFO][smbo.py:320] Finished 100 trials.
     [INFO][smbo.py:328] Configuration budget is exhausted:
-    [INFO][smbo.py:329] --- Remaining wallclock time: -3.065371036529541
+    [INFO][smbo.py:329] --- Remaining wallclock time: -3.8923418521881104
     [INFO][smbo.py:330] --- Remaining cpu time: inf
     [INFO][smbo.py:331] --- Remaining trials: 139
     Default cost (SuccessiveHalving): 0.36672856700711853
@@ -78,7 +84,7 @@ is chosen to optimize the average accuracy on 5-fold cross validation.
     [INFO][abstract_facade.py:203] Workers are reduced to 8.
     /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
     Perhaps you already have a cluster running?
-    Hosting the HTTP server on port 34681 instead
+    Hosting the HTTP server on port 39027 instead
       warnings.warn(
     [INFO][abstract_initial_design.py:147] Using 5 initial design configurations and 0 additional configurations.
     [INFO][successive_halving.py:164] Successive Halving uses budget type BUDGETS with eta 3, min budget 1, and max budget 25.
@@ -100,15 +106,15 @@ is chosen to optimize the average accuracy on 5-fold cross validation.
     [INFO][smbo.py:320] Finished 0 trials.
     [INFO][abstract_intensifier.py:516] Added config 6ad6db as new incumbent because there are no incumbents yet.
     [INFO][smbo.py:320] Finished 50 trials.
-    [INFO][abstract_intensifier.py:595] Added config da3ddf and rejected config 6ad6db as incumbent because it is not better than the incumbents on 1 instances:
-    [INFO][smbo.py:320] Finished 100 trials.
-    [INFO][abstract_intensifier.py:595] Added config 7b9b77 and rejected config da3ddf as incumbent because it is not better than the incumbents on 1 instances:
+    [INFO][abstract_intensifier.py:595] Added config c36dba and rejected config 6ad6db as incumbent because it is not better than the incumbents on 1 instances:
+    [INFO][abstract_intensifier.py:595] Added config 00a2dc and rejected config c36dba as incumbent because it is not better than the incumbents on 1 instances:
+    [INFO][abstract_intensifier.py:595] Added config df87dc and rejected config 00a2dc as incumbent because it is not better than the incumbents on 1 instances:
     [INFO][smbo.py:328] Configuration budget is exhausted:
-    [INFO][smbo.py:329] --- Remaining wallclock time: -0.8539242744445801
+    [INFO][smbo.py:329] --- Remaining wallclock time: -1.5701942443847656
     [INFO][smbo.py:330] --- Remaining cpu time: inf
-    [INFO][smbo.py:331] --- Remaining trials: 201
+    [INFO][smbo.py:331] --- Remaining trials: 159
     Default cost (Hyperband): 0.36672856700711853
-    Incumbent cost (Hyperband): 0.031168368926029033
+    Incumbent cost (Hyperband): 0.028935314144227897
 
 
 
@@ -287,7 +293,7 @@ is chosen to optimize the average accuracy on 5-fold cross validation.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (2 minutes 14.380 seconds)
+   **Total running time of the script:** (2 minutes 19.311 seconds)
 
 
 .. _sphx_glr_download_examples_2_multi_fidelity_1_mlp_epochs.py:

development/_sources/examples/2_multi_fidelity/2_sgd_datasets.rst.txt

@@ -66,14 +66,12 @@ the target function now is required to have an instance argument.
     [INFO][abstract_intensifier.py:595] Added config 8ce35d and rejected config 2b0dc0 as incumbent because it is not better than the incumbents on 45 instances:
     [INFO][smbo.py:320] Finished 400 trials.
     [INFO][abstract_intensifier.py:595] Added config 25625b and rejected config 8ce35d as incumbent because it is not better than the incumbents on 45 instances:
-    [INFO][abstract_intensifier.py:595] Added config 2cdaac and rejected config 25625b as incumbent because it is not better than the incumbents on 45 instances:
-    [INFO][smbo.py:320] Finished 450 trials.
     [INFO][smbo.py:328] Configuration budget is exhausted:
-    [INFO][smbo.py:329] --- Remaining wallclock time: -0.060822248458862305
+    [INFO][smbo.py:329] --- Remaining wallclock time: -0.07345461845397949
     [INFO][smbo.py:330] --- Remaining cpu time: inf
-    [INFO][smbo.py:331] --- Remaining trials: 4539
+    [INFO][smbo.py:331] --- Remaining trials: 4556
     Default cost: 0.15489347419148672
-    Incumbent cost: 0.00426383387912454
+    Incumbent cost: 0.004324883940174601
 
 
 
@@ -213,7 +211,7 @@ the target function now is required to have an instance argument.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 34.157 seconds)
+   **Total running time of the script:** (0 minutes 34.287 seconds)
 
 
 .. _sphx_glr_download_examples_2_multi_fidelity_2_sgd_datasets.py:

development/_sources/examples/2_multi_fidelity/3_specify_HB_via_total_budget.rst.txt

@@ -53,7 +53,7 @@ in terms of the fidelity units and let SMAC calculate how many trials that would
             max budget per trial:                    500
             total number of trials per HB round:     69
             budget used per HB round:                2629.6296296296296
-            number of brackets:                      3
+            number of brackets:                      4
             budgets per stage:                       {0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0], 1: [55.55555555555555, 166.66666666666666, 500.0], 2: [166.66666666666666, 500.0], 3: [500.0]}
             n configs per stage:                     {0: [27, 9, 3, 1], 1: [12, 4, 1], 2: [6, 2], 3: [4]}
     -----------------------------------------------------------------------------------------
@@ -206,7 +206,7 @@ in terms of the fidelity units and let SMAC calculate how many trials that would
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 7.835 seconds)
+   **Total running time of the script:** (0 minutes 8.016 seconds)
 
 
 .. _sphx_glr_download_examples_2_multi_fidelity_3_specify_HB_via_total_budget.py:

development/_sources/examples/2_multi_fidelity/sg_execution_times.rst.txt

@@ -6,7 +6,7 @@
 
 Computation times
 =================
-**02:56.371** total execution time for 3 files **from examples/2_multi_fidelity**:
+**03:01.615** total execution time for 3 files **from examples/2_multi_fidelity**:
 
 .. container::
 
@@ -33,11 +33,11 @@ Computation times
      - Time
      - Mem (MB)
    * - :ref:`sphx_glr_examples_2_multi_fidelity_1_mlp_epochs.py` (``1_mlp_epochs.py``)
-     - 02:14.380
+     - 02:19.311
      - 0.0
    * - :ref:`sphx_glr_examples_2_multi_fidelity_2_sgd_datasets.py` (``2_sgd_datasets.py``)
-     - 00:34.157
+     - 00:34.287
      - 0.0
    * - :ref:`sphx_glr_examples_2_multi_fidelity_3_specify_HB_via_total_budget.py` (``3_specify_HB_via_total_budget.py``)
-     - 00:07.835
+     - 00:08.016
      - 0.0

development/_sources/examples/3_multi_objective/1_schaffer.rst.txt

@@ -464,7 +464,7 @@ SMAC prioritizes the second objective over the first one.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 8.263 seconds)
+   **Total running time of the script:** (0 minutes 8.438 seconds)
 
 
 .. _sphx_glr_download_examples_3_multi_objective_1_schaffer.py:

development/_sources/examples/3_multi_objective/2_parego.rst.txt

@@ -47,40 +47,17 @@ mean accuracy and run-time of each configuration.
     [INFO][abstract_intensifier.py:516] Added config 71813f as new incumbent because there are no incumbents yet.
     [INFO][abstract_intensifier.py:603] Config bb3711 is a new incumbent. Total number of incumbents: 2.
     [INFO][abstract_intensifier.py:603] Config f07b8c is a new incumbent. Total number of incumbents: 3.
-    [INFO][abstract_intensifier.py:595] Added config 029a3c and rejected config 281184 as incumbent because it is not better than the incumbents on 2 instances:
-    [INFO][abstract_intensifier.py:603] Config 060aeb is a new incumbent. Total number of incumbents: 3.
-    [INFO][abstract_intensifier.py:603] Config ca383c is a new incumbent. Total number of incumbents: 4.
-    [INFO][abstract_intensifier.py:603] Config cc2c2b is a new incumbent. Total number of incumbents: 5.
-    [INFO][abstract_intensifier.py:603] Config 2ce7d9 is a new incumbent. Total number of incumbents: 6.
-    [INFO][abstract_intensifier.py:603] Config b02e1e is a new incumbent. Total number of incumbents: 7.
-    [INFO][abstract_intensifier.py:603] Config f691a7 is a new incumbent. Total number of incumbents: 8.
-    [INFO][abstract_intensifier.py:603] Config de75da is a new incumbent. Total number of incumbents: 9.
-    [INFO][abstract_intensifier.py:603] Config 20bbb6 is a new incumbent. Total number of incumbents: 10.
-    [INFO][abstract_intensifier.py:595] Added config 7780fd and rejected config 20bbb6 as incumbent because it is not better than the incumbents on 2 instances:
-    [INFO][abstract_intensifier.py:595] Added config 6233ce and rejected config f691a7 as incumbent because it is not better than the incumbents on 2 instances:
-    [INFO][abstract_intensifier.py:603] Config 11acfc is a new incumbent. Total number of incumbents: 11.
-    [INFO][abstract_intensifier.py:628] Removed one incumbent using crowding distance because more than 10 are available.
-    [INFO][smbo.py:320] Finished 50 trials.
-    [INFO][abstract_intensifier.py:603] Config c00065 is a new incumbent. Total number of incumbents: 11.
-    [INFO][abstract_intensifier.py:628] Removed one incumbent using crowding distance because more than 10 are available.
     [INFO][smbo.py:328] Configuration budget is exhausted:
-    [INFO][smbo.py:329] --- Remaining wallclock time: -0.15232491493225098
+    [INFO][smbo.py:329] --- Remaining wallclock time: -2.3799731731414795
     [INFO][smbo.py:330] --- Remaining cpu time: inf
-    [INFO][smbo.py:331] --- Remaining trials: 148
+    [INFO][smbo.py:331] --- Remaining trials: 182
     Validated costs from default config: 
-    --- [0.60155834 0.15452373]
+    --- [0.60155834 0.16958845]
 
     Validated costs from the Pareto front (incumbents):
-    --- [0.05092309 0.74322581]
-    --- [0.60407768 0.10240269]
-    --- [0.08652894 0.18247414]
-    --- [0.2490243  0.11120093]
-    --- [0.44185701 0.10148609]
-    --- [0.41235995 0.11730111]
-    --- [0.1964415  0.12692523]
-    --- [0.14051996 0.13725793]
-    --- [0.09626973 0.15918124]
-    --- [0.12104457 0.15442801]
+    --- [0.72761606 0.20369637]
+    --- [0.05092309 0.74873209]
+    --- [0.53243036 0.60367513]
 
 
 
@@ -261,7 +238,7 @@ mean accuracy and run-time of each configuration.
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 34.524 seconds)
+   **Total running time of the script:** (0 minutes 35.992 seconds)
 
 
 .. _sphx_glr_download_examples_3_multi_objective_2_parego.py:

development/_sources/examples/3_multi_objective/sg_execution_times.rst.txt

@@ -6,7 +6,7 @@
 
 Computation times
 =================
-**00:42.787** total execution time for 2 files **from examples/3_multi_objective**:
+**00:44.430** total execution time for 2 files **from examples/3_multi_objective**:
 
 .. container::
 
@@ -33,8 +33,8 @@ Computation times
      - Time
      - Mem (MB)
    * - :ref:`sphx_glr_examples_3_multi_objective_2_parego.py` (``2_parego.py``)
-     - 00:34.524
+     - 00:35.992
      - 0.0
    * - :ref:`sphx_glr_examples_3_multi_objective_1_schaffer.py` (``1_schaffer.py``)
-     - 00:08.263
+     - 00:08.438
      - 0.0

development/_sources/examples/5_commandline/1_call_target_function_script.rst.txt

@@ -112,7 +112,7 @@ This simple example shows how to call a script with the following content:
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** (0 minutes 19.297 seconds)
+   **Total running time of the script:** (0 minutes 19.839 seconds)
 
 
 .. _sphx_glr_download_examples_5_commandline_1_call_target_function_script.py:

development/_sources/examples/5_commandline/sg_execution_times.rst.txt

@@ -6,7 +6,7 @@
 
 Computation times
 =================
-**00:19.297** total execution time for 1 file **from examples/5_commandline**:
+**00:19.839** total execution time for 1 file **from examples/5_commandline**:
 
 .. container::
 
@@ -33,5 +33,5 @@ Computation times
      - Time
      - Mem (MB)
    * - :ref:`sphx_glr_examples_5_commandline_1_call_target_function_script.py` (``1_call_target_function_script.py``)
-     - 00:19.297
+     - 00:19.839
      - 0.0