Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request: tune_bayes() verbose options #677

Closed
cphaarmeyer opened this issue Apr 24, 2023 · 2 comments · Fixed by #682
Closed

Feature Request: tune_bayes() verbose options #677

cphaarmeyer opened this issue Apr 24, 2023 · 2 comments · Fixed by #682
Labels
bug an unexpected problem or unintended behavior

Comments

@cphaarmeyer
Copy link

tune_bayes() verbose options

In {tune} 1.0.0 verbose = TRUE resulted in:

library(parsnip)
library(rsample)
library(tune)
set.seed(42)

res <- tune_bayes(
  nearest_neighbor("regression", neighbors = tune()),
  mpg ~ .,
  vfold_cv(mtcars, v = 3),
  control = control_bayes(verbose = TRUE),
  iter = 2, initial = 3
)
#> 
#> ❯  Generating a set of 3 initial parameter results
#> ✓ Initialization complete
#> 
#> Optimizing rmse using the expected improvement
#> 
#> ── Iteration 1 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=3.387 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 12 candidates
#> i Predicted candidates
#> i neighbors=12
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=3.4 (+/-0.619)
#> 
#> ── Iteration 2 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=3.387 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 11 candidates
#> i Predicted candidates
#> i neighbors=9
#> i Estimating performance
#> ✓ Estimating performance
#> ♥ Newest results:    rmse=3.375 (+/-0.501)

Created on 2023-04-24 with reprex v2.0.2

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.3 (2023-03-15 ucrt)
#>  os       Windows 10 x64 (build 19045)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-04-24
#>  pandoc   2.19.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  class          7.3-21     2023-01-23 [2] CRAN (R 4.2.3)
#>  cli            3.6.1      2023-03-23 [1] CRAN (R 4.2.3)
#>  codetools      0.2-19     2023-02-01 [2] CRAN (R 4.2.3)
#>  colorspace     2.1-0      2023-01-23 [1] CRAN (R 4.2.2)
#>  data.table     1.14.8     2023-02-17 [1] CRAN (R 4.2.2)
#>  dials          1.2.0      2023-04-03 [1] CRAN (R 4.2.3)
#>  DiceDesign     1.9        2021-02-13 [1] CRAN (R 4.2.0)
#>  digest         0.6.31     2022-12-11 [1] CRAN (R 4.2.2)
#>  dplyr          1.1.1      2023-03-22 [1] CRAN (R 4.2.3)
#>  evaluate       0.20       2023-01-17 [1] CRAN (R 4.2.2)
#>  fansi          1.0.4      2023-01-22 [1] CRAN (R 4.2.2)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.2.2)
#>  foreach        1.5.2      2022-02-02 [1] CRAN (R 4.2.0)
#>  fs             1.6.1      2023-02-06 [1] CRAN (R 4.2.2)
#>  furrr          0.3.1      2022-08-15 [1] CRAN (R 4.2.1)
#>  future         1.32.0     2023-03-07 [1] CRAN (R 4.2.2)
#>  future.apply   1.10.0     2022-11-05 [1] CRAN (R 4.2.2)
#>  generics       0.1.3      2022-07-05 [1] CRAN (R 4.2.1)
#>  ggplot2        3.4.2      2023-04-03 [1] CRAN (R 4.2.3)
#>  globals        0.16.2     2022-11-21 [1] CRAN (R 4.2.2)
#>  glue           1.6.2      2022-02-24 [1] CRAN (R 4.2.0)
#>  gower          1.0.1      2022-12-22 [1] CRAN (R 4.2.2)
#>  GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.2.0)
#>  gtable         0.3.3      2023-03-21 [1] CRAN (R 4.2.3)
#>  hardhat        1.3.0      2023-03-30 [1] CRAN (R 4.2.3)
#>  htmltools      0.5.5      2023-03-23 [1] CRAN (R 4.2.3)
#>  igraph         1.4.2      2023-04-07 [1] CRAN (R 4.2.3)
#>  ipred          0.9-14     2023-03-09 [1] CRAN (R 4.2.2)
#>  iterators      1.0.14     2022-02-05 [1] CRAN (R 4.2.0)
#>  kknn         * 1.3.1      2016-03-26 [1] CRAN (R 4.2.3)
#>  knitr          1.42       2023-01-25 [1] CRAN (R 4.2.2)
#>  lattice        0.20-45    2021-09-22 [2] CRAN (R 4.2.3)
#>  lava           1.7.2.1    2023-02-27 [1] CRAN (R 4.2.2)
#>  lhs            1.1.6      2022-12-17 [1] CRAN (R 4.2.2)
#>  lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.2.1)
#>  listenv        0.9.0      2022-12-16 [1] CRAN (R 4.2.2)
#>  lubridate      1.9.2      2023-02-10 [1] CRAN (R 4.2.2)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.2.0)
#>  MASS           7.3-58.2   2023-01-23 [2] CRAN (R 4.2.3)
#>  Matrix         1.5-3      2022-11-11 [2] CRAN (R 4.2.3)
#>  modelenv       0.1.1      2023-03-08 [1] CRAN (R 4.2.2)
#>  munsell        0.5.0      2018-06-12 [1] CRAN (R 4.2.0)
#>  nnet           7.3-18     2022-09-28 [2] CRAN (R 4.2.3)
#>  parallelly     1.35.0     2023-03-23 [1] CRAN (R 4.2.3)
#>  parsnip      * 1.1.0      2023-04-12 [1] CRAN (R 4.2.3)
#>  pillar         1.9.0      2023-03-22 [1] CRAN (R 4.2.3)
#>  pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.2.0)
#>  prodlim        2023.03.31 2023-04-02 [1] CRAN (R 4.2.3)
#>  purrr          1.0.1      2023-01-10 [1] CRAN (R 4.2.2)
#>  R.cache        0.16.0     2022-07-21 [1] CRAN (R 4.2.1)
#>  R.methodsS3    1.8.2      2022-06-13 [1] CRAN (R 4.2.0)
#>  R.oo           1.25.0     2022-06-12 [1] CRAN (R 4.2.0)
#>  R.utils        2.12.2     2022-11-11 [1] CRAN (R 4.2.2)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.2.0)
#>  Rcpp           1.0.10     2023-01-22 [1] CRAN (R 4.2.2)
#>  recipes        1.0.5      2023-02-20 [1] CRAN (R 4.2.2)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.2.1)
#>  rlang          1.1.0      2023-03-14 [1] CRAN (R 4.2.2)
#>  rmarkdown      2.21       2023-03-26 [1] CRAN (R 4.2.3)
#>  rpart          4.1.19     2022-10-21 [2] CRAN (R 4.2.3)
#>  rsample      * 1.1.1      2022-12-07 [1] CRAN (R 4.2.2)
#>  rstudioapi     0.14       2022-08-22 [1] CRAN (R 4.2.1)
#>  scales         1.2.1      2022-08-20 [1] CRAN (R 4.2.1)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.2.0)
#>  styler         1.9.1      2023-03-04 [1] CRAN (R 4.2.2)
#>  survival       3.5-3      2023-02-12 [2] CRAN (R 4.2.3)
#>  tibble         3.2.1      2023-03-20 [1] CRAN (R 4.2.3)
#>  tidyr          1.3.0      2023-01-24 [1] CRAN (R 4.2.2)
#>  tidyselect     1.2.0      2022-10-10 [1] CRAN (R 4.2.1)
#>  timechange     0.2.0      2023-01-11 [1] CRAN (R 4.2.2)
#>  timeDate       4022.108   2023-01-07 [1] CRAN (R 4.2.2)
#>  tune         * 1.0.0      2022-07-07 [1] CRAN (R 4.2.3)
#>  utf8           1.2.3      2023-01-31 [1] CRAN (R 4.2.2)
#>  vctrs          0.6.2      2023-04-19 [1] CRAN (R 4.2.3)
#>  withr          2.5.0      2022-03-03 [1] CRAN (R 4.2.0)
#>  workflows      1.1.3      2023-02-22 [1] CRAN (R 4.2.2)
#>  xfun           0.38       2023-03-24 [1] CRAN (R 4.2.3)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.2.2)
#>  yardstick      1.1.0      2022-09-07 [1] CRAN (R 4.2.1)
#> 
#>  [1] C:/Users/philipp_h/AppData/Local/R/win-library/4.2
#>  [2] C:/Program Files/R/R-4.2.3/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

After reading the changelog I expected that verbose_iter replaced this option (like in finetune::control_sim_anneal()).
But now with {tune} 1.1.1 I get:

library(parsnip)
library(rsample)
library(tune)
set.seed(42)

res <- tune_bayes(
  nearest_neighbor("regression", neighbors = tune()),
  mpg ~ .,
  vfold_cv(mtcars, v = 3),
  control = control_bayes(verbose_iter = TRUE),
  iter = 2, initial = 3
)
#> Optimizing rmse using the expected improvement

Created on 2023-04-24 with reprex v2.0.2

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.3 (2023-03-15 ucrt)
#>  os       Windows 10 x64 (build 19045)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-04-24
#>  pandoc   2.19.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  class          7.3-21     2023-01-23 [2] CRAN (R 4.2.3)
#>  cli            3.6.1      2023-03-23 [1] CRAN (R 4.2.3)
#>  codetools      0.2-19     2023-02-01 [2] CRAN (R 4.2.3)
#>  colorspace     2.1-0      2023-01-23 [1] CRAN (R 4.2.2)
#>  data.table     1.14.8     2023-02-17 [1] CRAN (R 4.2.2)
#>  dials          1.2.0      2023-04-03 [1] CRAN (R 4.2.3)
#>  DiceDesign     1.9        2021-02-13 [1] CRAN (R 4.2.0)
#>  digest         0.6.31     2022-12-11 [1] CRAN (R 4.2.2)
#>  dplyr          1.1.1      2023-03-22 [1] CRAN (R 4.2.3)
#>  evaluate       0.20       2023-01-17 [1] CRAN (R 4.2.2)
#>  fansi          1.0.4      2023-01-22 [1] CRAN (R 4.2.2)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.2.2)
#>  foreach        1.5.2      2022-02-02 [1] CRAN (R 4.2.0)
#>  fs             1.6.1      2023-02-06 [1] CRAN (R 4.2.2)
#>  furrr          0.3.1      2022-08-15 [1] CRAN (R 4.2.1)
#>  future         1.32.0     2023-03-07 [1] CRAN (R 4.2.2)
#>  future.apply   1.10.0     2022-11-05 [1] CRAN (R 4.2.2)
#>  generics       0.1.3      2022-07-05 [1] CRAN (R 4.2.1)
#>  ggplot2        3.4.2      2023-04-03 [1] CRAN (R 4.2.3)
#>  globals        0.16.2     2022-11-21 [1] CRAN (R 4.2.2)
#>  glue           1.6.2      2022-02-24 [1] CRAN (R 4.2.0)
#>  gower          1.0.1      2022-12-22 [1] CRAN (R 4.2.2)
#>  GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.2.0)
#>  gtable         0.3.3      2023-03-21 [1] CRAN (R 4.2.3)
#>  hardhat        1.3.0      2023-03-30 [1] CRAN (R 4.2.3)
#>  htmltools      0.5.5      2023-03-23 [1] CRAN (R 4.2.3)
#>  igraph         1.4.2      2023-04-07 [1] CRAN (R 4.2.3)
#>  ipred          0.9-14     2023-03-09 [1] CRAN (R 4.2.2)
#>  iterators      1.0.14     2022-02-05 [1] CRAN (R 4.2.0)
#>  kknn         * 1.3.1      2016-03-26 [1] CRAN (R 4.2.3)
#>  knitr          1.42       2023-01-25 [1] CRAN (R 4.2.2)
#>  lattice        0.20-45    2021-09-22 [2] CRAN (R 4.2.3)
#>  lava           1.7.2.1    2023-02-27 [1] CRAN (R 4.2.2)
#>  lhs            1.1.6      2022-12-17 [1] CRAN (R 4.2.2)
#>  lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.2.1)
#>  listenv        0.9.0      2022-12-16 [1] CRAN (R 4.2.2)
#>  lubridate      1.9.2      2023-02-10 [1] CRAN (R 4.2.2)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.2.0)
#>  MASS           7.3-58.2   2023-01-23 [2] CRAN (R 4.2.3)
#>  Matrix         1.5-3      2022-11-11 [2] CRAN (R 4.2.3)
#>  modelenv       0.1.1      2023-03-08 [1] CRAN (R 4.2.2)
#>  munsell        0.5.0      2018-06-12 [1] CRAN (R 4.2.0)
#>  nnet           7.3-18     2022-09-28 [2] CRAN (R 4.2.3)
#>  parallelly     1.35.0     2023-03-23 [1] CRAN (R 4.2.3)
#>  parsnip      * 1.1.0      2023-04-12 [1] CRAN (R 4.2.3)
#>  pillar         1.9.0      2023-03-22 [1] CRAN (R 4.2.3)
#>  pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.2.0)
#>  prodlim        2023.03.31 2023-04-02 [1] CRAN (R 4.2.3)
#>  purrr          1.0.1      2023-01-10 [1] CRAN (R 4.2.2)
#>  R.cache        0.16.0     2022-07-21 [1] CRAN (R 4.2.1)
#>  R.methodsS3    1.8.2      2022-06-13 [1] CRAN (R 4.2.0)
#>  R.oo           1.25.0     2022-06-12 [1] CRAN (R 4.2.0)
#>  R.utils        2.12.2     2022-11-11 [1] CRAN (R 4.2.2)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.2.0)
#>  Rcpp           1.0.10     2023-01-22 [1] CRAN (R 4.2.2)
#>  recipes        1.0.5      2023-02-20 [1] CRAN (R 4.2.2)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.2.1)
#>  rlang          1.1.0      2023-03-14 [1] CRAN (R 4.2.2)
#>  rmarkdown      2.21       2023-03-26 [1] CRAN (R 4.2.3)
#>  rpart          4.1.19     2022-10-21 [2] CRAN (R 4.2.3)
#>  rsample      * 1.1.1      2022-12-07 [1] CRAN (R 4.2.2)
#>  rstudioapi     0.14       2022-08-22 [1] CRAN (R 4.2.1)
#>  scales         1.2.1      2022-08-20 [1] CRAN (R 4.2.1)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.2.0)
#>  styler         1.9.1      2023-03-04 [1] CRAN (R 4.2.2)
#>  survival       3.5-3      2023-02-12 [2] CRAN (R 4.2.3)
#>  tibble         3.2.1      2023-03-20 [1] CRAN (R 4.2.3)
#>  tidyr          1.3.0      2023-01-24 [1] CRAN (R 4.2.2)
#>  tidyselect     1.2.0      2022-10-10 [1] CRAN (R 4.2.1)
#>  timechange     0.2.0      2023-01-11 [1] CRAN (R 4.2.2)
#>  timeDate       4022.108   2023-01-07 [1] CRAN (R 4.2.2)
#>  tune         * 1.1.1      2023-04-11 [1] CRAN (R 4.2.3)
#>  utf8           1.2.3      2023-01-31 [1] CRAN (R 4.2.2)
#>  vctrs          0.6.2      2023-04-19 [1] CRAN (R 4.2.3)
#>  withr          2.5.0      2022-03-03 [1] CRAN (R 4.2.0)
#>  workflows      1.1.3      2023-02-22 [1] CRAN (R 4.2.2)
#>  xfun           0.38       2023-03-24 [1] CRAN (R 4.2.3)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.2.2)
#>  yardstick      1.1.0      2022-09-07 [1] CRAN (R 4.2.1)
#> 
#>  [1] C:/Users/philipp_h/AppData/Local/R/win-library/4.2
#>  [2] C:/Program Files/R/R-4.2.3/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Can the old behaviour be restored somehow?
@simonpcouch
Copy link
Contributor

simonpcouch commented Apr 24, 2023 •
edited
Loading

Ah, I see. Thanks for the issue! The difference between those arguments has been obfuscated since 1.0.0. For the meantime before we get a fix out, you can still get that verbose_iter output a la 1.0.0 by setting verbose = TRUE as before:

library(parsnip)
library(rsample)
library(tune)
set.seed(42)

res <- tune_bayes(
  nearest_neighbor("regression", neighbors = tune()),
  mpg ~ .,
  vfold_cv(mtcars, v = 3),
  control = control_bayes(verbose = TRUE),
  iter = 2, initial = 3
)
#> 
#> ❯  Generating a set of 3 initial parameter results
#> ✓ Initialization complete
#> 
#> 
#> ── Iteration 1 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=3.557 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 12 candidates
#> i Predicted candidates
#> i neighbors=12
#> i Estimating performance
#> i Fold1: preprocessor 1/1
#> ✓ Fold1: preprocessor 1/1
#> i Fold1: preprocessor 1/1, model 1/1
#> ✓ Fold1: preprocessor 1/1, model 1/1
#> i Fold1: preprocessor 1/1, model 1/1 (extracts)
#> i Fold1: preprocessor 1/1, model 1/1 (predictions)
#> i Fold2: preprocessor 1/1
#> ✓ Fold2: preprocessor 1/1
#> i Fold2: preprocessor 1/1, model 1/1
#> ✓ Fold2: preprocessor 1/1, model 1/1
#> i Fold2: preprocessor 1/1, model 1/1 (extracts)
#> i Fold2: preprocessor 1/1, model 1/1 (predictions)
#> i Fold3: preprocessor 1/1
#> ✓ Fold3: preprocessor 1/1
#> i Fold3: preprocessor 1/1, model 1/1
#> ✓ Fold3: preprocessor 1/1, model 1/1
#> i Fold3: preprocessor 1/1, model 1/1 (extracts)
#> i Fold3: preprocessor 1/1, model 1/1 (predictions)
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=3.572 (+/-0.861)
#> 
#> ── Iteration 2 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=3.557 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 11 candidates
#> i Predicted candidates
#> i neighbors=9
#> i Estimating performance
#> i Fold1: preprocessor 1/1
#> ✓ Fold1: preprocessor 1/1
#> i Fold1: preprocessor 1/1, model 1/1
#> ✓ Fold1: preprocessor 1/1, model 1/1
#> i Fold1: preprocessor 1/1, model 1/1 (extracts)
#> i Fold1: preprocessor 1/1, model 1/1 (predictions)
#> i Fold2: preprocessor 1/1
#> ✓ Fold2: preprocessor 1/1
#> i Fold2: preprocessor 1/1, model 1/1
#> ✓ Fold2: preprocessor 1/1, model 1/1
#> i Fold2: preprocessor 1/1, model 1/1 (extracts)
#> i Fold2: preprocessor 1/1, model 1/1 (predictions)
#> i Fold3: preprocessor 1/1
#> ✓ Fold3: preprocessor 1/1
#> i Fold3: preprocessor 1/1, model 1/1
#> ✓ Fold3: preprocessor 1/1, model 1/1
#> i Fold3: preprocessor 1/1, model 1/1 (extracts)
#> i Fold3: preprocessor 1/1, model 1/1 (predictions)
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=3.565 (+/-0.794)

Created on 2023-04-24 with reprex v2.0.2

The verbose_iter output will just be mixed in with verbose messages.

@simonpcouch simonpcouch added the bug an unexpected problem or unintended behavior label Apr 24, 2023
@simonpcouch simonpcouch mentioned this issue May 3, 2023
Merged
@github-actions
Copy link

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions bot locked and limited conversation to collaborators May 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

align verbosity options with documented behavior tidymodels/tune
2 participants
@simonpcouch @cphaarmeyer