pucAggregate() breaks REMIND input data generation

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

I have four preprocesing-remind runs that all fail on pucAggregate() because renv can't find the sources for some CRAN package:

• /p/tmp/pehl/preprocessing-remind/log-23667536.out
• /p/tmp/pehl/preprocessing-remind/log-23668164.out
• /p/tmp/pehl/preprocessing-remind/log-23668216.out
• /p/tmp/pehl/preprocessing-remind/log-23668267.out

Example output:

$ tail -n 50 /p/tmp/pehl/preprocessing-remind/log-23668267.out
Local configuration update:
  regionmapping: regionmappingH12.csv -> regionmapping_21_EU11.csv
 - data will be created from existing puc (rev6.305002_remind.puc).
Run pucAggregate(puc = "/p/projects/rd3mod/inputdata/puc/rev6.305002_remind.puc", regionmapping = "regionmapping_21_EU11.csv")

Error: <callr_status_error: callr subprocess failed: failed to find source for 'KernSmooth 2.23-18' in package repositories>
-->
<callr_remote_error in NULL:
 failed to find source for 'KernSmooth 2.23-18' in package repositories>
 in process 3497 

 Stack trace:

 Process 3000:
 1. madrat:::retrieveData(model = "REMIND", regionmapping = mapping[["regionmap ...
 2. base:::do.call(pucAggregate, c(list(puc = file.path(getConfig("pucfolder"), ...
 3. (function (puc, regionmapping = getConfig("regionmapping"), ...,  ...
 4. withr:::with_tempdir({ ...
 5. withr::with_dir(tmp, code)
 6. base:::force(code)
 7. utils:::capture.output(r(.aggregatePuc, list(regionmapping = regionmapping, ...
 8. utils:::evalVis(expr)
 9. base:::withVisible(eval(expr, pf))
 10. base:::eval(expr, pf)
 11. base:::eval(expr, pf)
 12. callr:::r(.aggregatePuc, list(regionmapping = regionmapping,  ...
 13. callr:::get_result(output = out, options)
 14. throw(newerr, parent = remerr[[2]])

 x callr subprocess failed: failed to find source for 'KernSmooth 2.23-18' in package repositories 

 Process 3497:
 26. (function (regionmapping, cfg, madratCfg, nestinglevel)  ...
 27. renv::restore(lockfile = "puc/renv.lock", prompt = FALSE)
 28. renv:::renv_restore_run_actions(project, diff, current, lockfile,  ...
 29. renv:::retrieve(packages)
 30. renv:::handler(package, renv_retrieve_impl(package))
 31. renv:::renv_retrieve_impl(package)
 32. renv:::renv_retrieve_repos(record)
 33. (function() { ...
 34. base:::warning(error)
 35. base:::withRestarts({ ...
 36. base:::withOneRestart(expr, restarts[[1L]])
 37. base:::doWithOneRestart(return(expr), restart)
 38. (function (e)  ...

 x failed to find source for 'KernSmooth 2.23-18' in package repositories 

There were 50 or more warnings (use warnings() to see the first 50)
Execution halted

The runs devtools::load_all() this mrremind commit for testing purposes.

Naturally, preprocessing crashes and no output is generated.

[tscheypidi]

@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q We are currently looking into this but have not understood yet why certain packages are missing in the renv cache (they cannot be download as the nodes you have the preprocessing running on do not have internet access).

As a workaround you can set renv = FALSE in retrieveData

[pfuehrlich-pik]

In addition to the Problem Jan mentioned we noticed that your renv.lock does not have the RSE server configured as a repo. This is probably not a problem if all packages are available in cache, but I'd still argue the renv.lock is broken if it cannot be restored on a system with internet access but no renv cache. The question is why is the RSE server not configured as a repo? Are you using the default global .Rprofile?

[pfuehrlich-pik]

The renv.lock I'm talking about is located in the .puc which is actually a .tgz that you can unpack as usual.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

The renv.lock I'm talking about is located in the .puc which is actually a .tgz that you can unpack as usual.

I disclaim all responsibility for files generated by madrat.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

/p/tmp/pehl/preprocessing-remind/log-23668267.out was created by R --vanilla.

The other three, which utilised /p/projects/rd3mod/inputdata/puc/rev6.305001_remind.puc used the default setup, including the RSE package repository.

$ tar -Oxzf /p/projects/rd3mod/inputdata/puc/rev6.305001_remind.puc ./renv.lock | grep -A 9 Repositories
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://cran.rstudio.com"
      },
      {
        "Name": "pik",
        "URL": "https://rse.pik-potsdam.de/r/packages"
      }
    ]

Same error.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Using the library mrremind version instead of using devtool::load_all() shows the same error. /p/tmp/pehl/preprocessing-remind/log-23676971.out