Skip on CRAN problems with 3.1.0?

[MarkEdmondson1234]

Hi,

I have today been emailed by CRAN about a failing test which failed suddenly after a few months being ok. I saw you just had a new release (https://github.com/r-lib/testthat/releases/tag/v3.1.0) so I'm flagging it up here in case it is related or can rule it out as a coincidence.

The failure seems to be related to test processes
https://github.com/MarkEdmondson1234/googleAnalyticsR/issues/389

On CRAN I mostly skip_on_cran() everything so it seems its related to R processes for each test thread not looking for env arguments in a consistent way which it was doing before the release.
https://github.com/MarkEdmondson1234/googleAnalyticsR/blob/master/tests/testthat/setup.R

Or perhaps skip_on_cran() is not working as expected? This test should not run as it will fail when no authentication, but now is:

test_that("Correct authentication", {
  skip_on_travis()
  skip_on_cran()
  
  al <- ga_account_list()
  expect_true(accountId %in% al$accountId)
  expect_true(accountId2 %in% al$accountId)
  expect_true(ga_id %in% al$viewId)
  expect_true(ga_id2 %in% al$viewId)
})

I've been told to fix it or else the package will be removed so my short term solution is remove all tests for a CRAN submission, but ideally I'd like to find the root cause and I guess you will have a lot of reports if it is related.

[hadley]

Link to actual failure: https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-gcc/googleAnalyticsR-00check.html and actual text:

  Error: testthat subprocess failed to start, stderr:
  ℹ 2021-10-04 23:15:27 > No consent file found
  Setting deferred event(s) on global environment.
    * Execute (and clear) with `withr::deferred_run()`.
    * Clear (without executing) with `withr::deferred_clear()`.
  Local auth with same test key
  No local auth - set GA_TEST_AUTH
  Error: Test failed
  
  Backtrace:
       █
    1. └─testthat::test_check("googleAnalyticsR")
    2. └─testthat::test_dir(...)
    3. └─testthat:::test_files(...)
    4. └─testthat:::test_files(...)
    5. ├─testthat::with_reporter(...)
    6. │ └─base::tryCatch(...)
    7. │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
    8. │ └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
    9. │ └─base:::doTryCatch(return(expr), name, parentenv, handler)
   10. └─testthat:::parallel_event_loop_chunky(queue, reporters)
   11. └─queue$poll(Inf)
   12. └─base::lapply(...)
   13. └─testthat:::FUN(X[[i]], ...)
   14. └─private$handle_error(msg, i)
  Execution halted

It's pretty suspicious that we see "Setting deferred event(s) on global environment", which suggests that error code is being run outside of test_that(). I can't imagine that I could've broken skip_on_cran() globally (or we'd see a very large number of CRAN failures), but it's possible that it's some interaction with skipping and parallel tests.

[MarkEdmondson1234]

I installed 3.1.0 testthat locally and ran the tests, it passed ok, although those are all not skipped as they are on CRAN. I had a look through the test files to see if there is anything outside of the test_that() functions but nothing found such as context() that I used to use.

For now I've put the tests folder in my ignore build file and submitted 1.0.1 to avoid CRAN removing it.

[hadley]

@rdiaz02 @FelixErnst that appears to be #1472

[rdiaz02]

Yes, you are right. Further details there.

[skyebend]

I'm having problems with this also. My package ndtv started failing tests on CRAN. I previously had an issue with these tests, and enclosed with skip_on_cran().

     + # only check MDSJ if allready installed (and not on CRAN)
     + skip_on_cran()
     + has.mdsj<-ndtv:::check.mdsj(ask=FALSE)
     + if(!is.null(has.mdsj)){
     + proximity.timeline(cls33_10_16_96,onsets=seq(0,45,5),termini=seq(2.5,47.5,5),mode='MDSJ')
     + }

https://github.com/statnet/ndtv/blob/689001ab01d3a50b82eef15719b0646b19496590/ndtv/tests/layout_tests.R#L156

This started failing again shortly after testthat release (sorry I don't have better details, I didn't get to it in time and ndtv package was taken down from CRAN) Maybe I have an incorrect use of skip_on_cran() in the test?

Looks like it is only failing for some linux flavors? (or could be failing more widely, but the side effects of my test only cause failures on CRAN's linux tests)
https://cran-archive.r-project.org/web/checks/2021/2021-10-22_check_results_ndtv.html

[skyebend]

Anything I can do to help debug this? I have two packages now failing CRAN tests

[gaborcsardi]

@MarkEdmondson1234 @skyebend It seems that you use test_that() and skip_on_cran() outside of test files (i.e. outside of tests/testthat/test-*.R), and although apparently this has worked before the latest release, it is not a file organization that we tested.

If you don't want to wait for the fix in testthat, you can move your test_that() blocks into a tests/testthat/test-*.R file and that should fix this.

[wlandau]

In both testthat 3.1.4 and 9868764, I am seeing this issue even in tests inside tests/testthat/test-*.R. I noticed because a recent submission was rejected due to long check times.

library(testthat)
library(usethis)
tmp <- capture.output(suppressMessages(create_package("pkg", open = FALSE)))
setwd("pkg")
tmp <- suppressMessages(use_testthat())
line <- "test_that('x', {expect_equal(Sys.getenv('NOT_CRAN', unset = ''), '')})"
writeLines(line, "tests/testthat/test-envvar.R")
print(Sys.getenv("NOT_CRAN", unset = ""))
#> [1] ""
test_local(".", reporter = "Summary")
#> envvar: 1
#> 
#> ══ Failed ══════════════════════════════════════════════════════════════════════
#> ── 1. Failure (test-envvar.R:1:17): x ──────────────────────────────────────────
#> Sys.getenv("NOT_CRAN", unset = "") (`actual`) not equal to "" (`expected`).
#> 
#> `actual`:   "true"
#> `expected`: ""    
#> 
#> ══ DONE ════════════════════════════════════════════════════════════════════════
#> No one is perfect!
#> Error: Test failures

^{Created on 2022-09-15 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.1 (2022-06-23)
#>  os       macOS Big Sur ... 10.16
#>  system   x86_64, darwin17.0
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/Indiana/Indianapolis
#>  date     2022-09-15
#>  pandoc   2.18 @ /Applications/RStudio.app/Contents/MacOS/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version    date (UTC) lib source
#>    brio          1.1.3      2021-11-30 [1] CRAN (R 4.2.0)
#>    cli           3.3.0      2022-04-25 [1] CRAN (R 4.2.0)
#>    crayon        1.5.1      2022-03-26 [1] CRAN (R 4.2.0)
#>    desc          1.4.1      2022-03-06 [1] CRAN (R 4.2.0)
#>    diffobj       0.3.5      2021-10-05 [1] CRAN (R 4.2.0)
#>    digest        0.6.29     2021-12-01 [1] CRAN (R 4.2.0)
#>    evaluate      0.15       2022-02-18 [1] CRAN (R 4.2.0)
#>    fansi         1.0.3      2022-03-24 [1] CRAN (R 4.2.0)
#>    fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.2.0)
#>    fs            1.5.2      2021-12-08 [1] CRAN (R 4.2.0)
#>    glue          1.6.2      2022-02-24 [1] CRAN (R 4.2.0)
#>    highr         0.9        2021-04-16 [1] CRAN (R 4.2.0)
#>    htmltools     0.5.2      2021-08-25 [1] CRAN (R 4.2.0)
#>    knitr         1.39       2022-04-26 [1] CRAN (R 4.2.0)
#>    lifecycle     1.0.1.9000 2022-07-27 [1] Github (r-lib/lifecycle@5e11675)
#>    magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.2.0)
#>    pillar        1.8.0      2022-07-18 [1] CRAN (R 4.2.0)
#>  R pkg         * 0.0.0.9000 <NA>       [?] <NA>
#>    pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.2.0)
#>    pkgload       1.3.0      2022-06-27 [1] CRAN (R 4.2.0)
#>    purrr         0.3.4      2020-04-17 [1] CRAN (R 4.2.0)
#>    R.cache       0.15.0     2021-04-30 [1] CRAN (R 4.2.0)
#>    R.methodsS3   1.8.2      2022-06-13 [1] CRAN (R 4.2.0)
#>    R.oo          1.25.0     2022-06-12 [1] CRAN (R 4.2.0)
#>    R.utils       2.11.0     2021-09-26 [1] CRAN (R 4.2.0)
#>    R6            2.5.1      2021-08-19 [1] CRAN (R 4.2.0)
#>    rematch2      2.1.2      2020-05-01 [1] CRAN (R 4.2.0)
#>    reprex        2.0.1      2021-08-05 [1] CRAN (R 4.2.0)
#>    rlang         1.0.4      2022-07-12 [1] CRAN (R 4.2.0)
#>    rmarkdown     2.14       2022-04-25 [1] CRAN (R 4.2.0)
#>    roxygen2      7.2.1      2022-07-18 [1] CRAN (R 4.2.0)
#>    rprojroot     2.0.3      2022-04-02 [1] CRAN (R 4.2.0)
#>    rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.2.0)
#>    sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.2.0)
#>    stringi       1.7.8      2022-07-11 [1] CRAN (R 4.2.0)
#>    stringr       1.4.0      2019-02-10 [1] CRAN (R 4.2.0)
#>    styler        1.7.0      2022-03-13 [1] CRAN (R 4.2.0)
#>    testthat    * 3.1.4.9000 2022-09-15 [1] Github (r-lib/testthat@9868764)
#>    tibble        3.1.8      2022-07-22 [1] CRAN (R 4.2.0)
#>    usethis     * 2.1.6.9000 2022-07-27 [1] Github (r-lib/usethis@b0318bf)
#>    utf8          1.2.2      2021-07-24 [1] CRAN (R 4.2.0)
#>    vctrs         0.4.1      2022-04-13 [1] CRAN (R 4.2.0)
#>    waldo         0.4.0      2022-03-16 [1] CRAN (R 4.2.0)
#>    whisker       0.4        2019-08-28 [1] CRAN (R 4.2.0)
#>    withr         2.5.0      2022-03-03 [1] CRAN (R 4.2.0)
#>    xfun          0.31       2022-05-10 [1] CRAN (R 4.2.0)
#>    xml2          1.3.3      2021-11-30 [1] CRAN (R 4.2.0)
#>    yaml          2.3.5      2022-02-21 [1] CRAN (R 4.2.0)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.2/Resources/library
#> 
#>  R ── Package was removed from disk.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[hadley]

@wlandau I don't understand what you're trying to say — to me, your example looks like it proves that NOT_CRAN is being set correctly.

[wlandau]

On my end, Sys.getenv("NOT_CRAN") is "true" inside the test even though I set it to "" outside the test. I am trying to figure out if this is why the total check time of my last submission of targets was so high (i.e. if long tests ran on CRAN even though I wrote skip_on_cran() at the top).

[gaborcsardi]

testthat::test_local() sets NOT_CRAN to "true":

> testthat::test_local
function(path = ".", reporter = NULL, ...) {
  package <- pkgload::pkg_name(path)
  test_path <- file.path(pkgload::pkg_path(path), "tests", "testthat")

  withr::local_envvar(NOT_CRAN = "true")
  test_dir(
    test_path,
    package = package,
    reporter = reporter,
    ...,
    load_package = if (package != "testthat") "source" else "none"
  )
}

Don't use test_local() on CRAN, use test_check().

[wlandau]

testthat::test_local() sets NOT_CRAN to "true":

Hmm... so does devtools::check() apparently, even with test_check() in the default tests/testthat.R. That may explain what I was seeing. I think for my case it will be sufficient to use R CMD check directly.