Recipe rainfarm no longer working

[bouweandela]

When I try to run recipe_rainfarm.yml, the diagnostic script crashes with the following error message:

Julia version 1.0.3 at location /home/bandela/conda/envs/test/bin will be used.
Loading setup script for JuliaCall...
WARNING: could not import StatsModels.Formula into RCall
ERROR: LoadError: LoadError: UndefVarError: Formula not defined
Stacktrace:
 [1] top-level scope at none:0
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1044
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at /home/bandela/.julia/packages/RCall/iojZI/src/RCall.jl:2
 [6] top-level scope at none:0
 [7] include at ./boot.jl:317 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1044
 [9] include(::Module, ::String) at ./sysimg.jl:29
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:319 [inlined]
 [12] eval(::Expr) at ./client.jl:393
 [13] top-level scope at ./none:3
in expression starting at /home/bandela/.julia/packages/RCall/iojZI/src/convert/formula.jl:39
in expression starting at /home/bandela/.julia/packages/RCall/iojZI/src/RCall.jl:46
LoadError("/home/bandela/conda/envs/test/lib/R/library/JuliaCall/julia/setup.jl", 72, ErrorException("Failed to precompile RCall [6f49c342-dc21-5d91-9882-a32aef131414] to /home/bandela/.julia/compiled/v1.0/RCall/8GFyb.ji.")) Error in .julia$cmd(paste0("Base.include(Main,\"", system.file("julia/setup.jl",  : 
  Error happens when you try to execute command Base.include(Main,"/home/bandela/conda/envs/test/lib/R/library/JuliaCall/julia/setup.jl") in Julia.
                        To have more helpful error messages,
                        you could considering running the command in Julia directly
Calls: julia_setup -> <Anonymous>
Execution halted
Julia exit.

It looks like Formula was removed in version 0.6.0 of the StatsModels package, because installing version 0.5.0 solves the problem.

[jhardenberg]

This seems to be a bug in a specific Julia package (RCall) which is used in turn by the JuliaCall R library which we use. I found a description here JuliaInterop/RCall.jl#313 and here JuliaInterop/RCall.jl#30
Indeed for now the solution seems to be to open julia from the command line and to give the following commands:

]
add [email protected]
update RCall

There does not seem to be anything which we can really do from ESMValTool for now.
Actually what we should do (and I will work on it soon) is to rewrite the R wrapper of rainfarm entirely in Julia, now that ESMValTool actually supports it directly. This way there would be no need for using JuliaCall.

[valeriupredoi]

do not install julia from conda-forge -> that don't work no more, it spits out a ton of errors, instead do what CircleCI does:

wget https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.3-linux-x86_64.tar.gz
tar xfz julia-*-linux-x86_64.tar.gz
ln -s $PWD/julia-*/bin/julia $HOME/$ANACONDA/envs/esmvaltool/bin
which julia
julia --help
julia esmvaltool/install/Julia/setup.jl

softlinking Julia to the $ANACONDA bin is important so that the diagnostic picks up the executable; $ANACONDA is just your anaconda3 repo

[valeriupredoi]

also note a possible issue that I am encountering on Jasmin (and not on my machine): JuliaLang/julia#30640

In my case, elegantly solved by stuffing Julia's face with the registries: since I am behind an odd firewall on Jasmin that allows me to git clone I just shoved the registry in where it's supposed to begit clone https://github.com/JuliaRegistries/General.git ~/.julia/registries/General and then voila, it updated it correctly and installed my packages 🍺

[valeriupredoi]

I am actually gonna make a separate issue for this since we need to edit the documentation as well..

[valeriupredoi]

with the new sort of installing Julia, now I get this from rainfarm:

Julia version 1.0.3 at location /home/users/valeriu/julia-1.0.3/bin will be used.
Loading setup script for JuliaCall...
Finish loading setup script for JuliaCall.
[1] "rainfarm : starting routine"
[1] "rainfarm: calling rainfarm"
Error: Error happens in Julia.
NetCDF error code -101:
        NetCDF: HDF error
Stacktrace:
 [1] check at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_helpers.jl:22 [inlined]
 [2] nc_open(::String, ::UInt16, ::Array{Int32,1}) at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_c.jl:217
 [3] nc_open at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_helpers.jl:101 [inlined]
 [4] #open#41(::UInt16, ::Bool, ::Function, ::String) at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/NetCDF.jl:711
 [5] #open at ./none:0 [inlined]
 [6] read_netcdf2d(::String, ::String) at /home/users/valeriu/.julia/packages/RainFARM/JhJji/src/rf/read_netcdf2d.jl:9
 [7] docall(::Ptr{Nothing}) at /home/users/valeriu/anaconda3R/envs/esmvaltool/lib/R/library/JuliaCall/julia/setup.jl:168
Execution halted
Julia exit.

Any clues?

[valeriupredoi]

fairly sure it's coz of hdf5=1.10.5 not being supported by julia=1.0.3

[valeriupredoi]

bah no, I was severly wrong on that, I found what the issue is - it's solely related to RainFARM/read_netcdf2d:

> infile <- "/home/users/valeriu/esmvaltool_var_test/LEE/recipe_rainfarm_20190820_122102/preproc/rainfarm/pr/CMIP5_EC-EARTH_day_historical_r1i1p1_pr_1997-1997.nc"
> varname <- "pr"
> ans <- julia_call("read_netcdf2d", infile, varname, need_return = "R" )Error: Error happens in Julia.
NetCDF error code -101:
	NetCDF: HDF error
Stacktrace:
 [1] check at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_helpers.jl:22 [inlined]
 [2] nc_open(::String, ::UInt16, ::Array{Int32,1}) at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_c.jl:217
 [3] nc_open at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/netcdf_helpers.jl:101 [inlined]
 [4] #open#41(::UInt16, ::Bool, ::Function, ::String) at /home/users/valeriu/.julia/packages/NetCDF/wnKH9/src/NetCDF.jl:711
 [5] #open at ./none:0 [inlined]
 [6] read_netcdf2d(::String, ::String) at /home/users/valeriu/.julia/packages/RainFARM/JhJji/src/rf/read_netcdf2d.jl:9
 [7] docall(::Ptr{Nothing}) at /home/users/valeriu/anaconda3R/envs/esmvaltool/lib/R/library/JuliaCall/julia/setup.jl:168
> 

but that exact loading method works fine with, say, a normal OBS file -> more importantly it works fine with the original CMIP5 file before being preprocessed:

infile <- "/badc/cmip5/data/cmip5/output1/ICHEC/EC-EARTH/historical/day/atmos/day/r1i1p1/latest/pr/pr_day_EC-EARTH_historical_r1i1p1_19900101-19991231.nc"
varname <- "pr"
ans <- julia_call("read_netcdf2d", infile, varname, need_return = "R" )

interestingly enough, a pure julia load of the file works fine too

julia> using NetCDF

julia> nc = NetCDF.open("/home/users/valeriu/esmvaltool_var_test/LEE/recipe_rainfarm_20190820_122102/preproc/rainfarm/pr/CMIP5_EC-EARTH_day_historical_r1i1p1_pr_1997-1997.nc", mode=NC_NOWRITE)

so it's not a Julia error, it's something to do with the communication between Julia and R...

[valeriupredoi]

I'll leave it to you @jhardenberg to see what's going on, but I think @bouweandela 's original issue is sorted out by a correct installation of Julia with JuliaCall since that's working now 🍺

[jhardenberg]

I was not able to reproduce the error, since I actually run into a different issue.
After creating a new environment, reinstalling esmvaltool and installing julia as in #1254, the diagnostic runs into a new issue with JuliaCall segfaulting when called from R....
Possibly a side effect of using a more recent version of R in the environment?
Is it only me or can also somebody else reproduce this?
I actually wonder how much effort to put in investigating this further: adopting #1242 (the rainfarm diagnostic based on pure julia) there would be no need for JuliaCall at all ....

[valeriupredoi]

adopt #1242 by me, if it works fine and we need one less dependency, a win-win. The only question I can think of is if there are users that code in R only to call Julia from within their R environment (seems like a major pain in the arse to me but hey)

runs into a new issue with JuliaCall segfaulting when called from R....

dies it load the library and modules?

library(JuliaCall)
julia_setup()
julia_library("RainFARM")

[valeriupredoi]

the call to setup and loading the RainFARM library work for me with:

(esmvaltool) valeriu@valeriu-PORTEGE-Z30-C:~$ julia --version
julia version 1.0.3
(esmvaltool) valeriu@valeriu-PORTEGE-Z30-C:~$ conda list r-base
# packages in environment at /home/valeriu/anaconda3R/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
r-base                    3.6.1                h8900bf8_2    conda-forge
(esmvaltool) valeriu@valeriu-PORTEGE-Z30-C:~$ R --version
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

[jhardenberg]

dies it load the library and modules?
library(JuliaCall)
julia_setup()

Actually not. Opening R and calling those two commands crashes while doing julia_setup()
The versions of my packages are identical to yours. A mistery ...
Enclosed is a log of what I get .... strangely it seems to be using /usr/lib/R/lib/libR.so at the beginning even if later it uses the version from my environment /home/jost/miniconda3/envs/esmvaltool3/lib/R/bin/exec/../../lib/libR.so
juliacall_crash.log

In any case it may not be so important to investigate this further, since calling julia directly (instead of passing through R) should be preferrable, as implemented in #1242
In #1240 I also prepared an example template diagnostic in pure julia to be used for future development in julia.

[valeriupredoi]

Totally with you on this one, Jost 🍺 Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Thu, 22 Aug 2019, 17:17 Jost von Hardenberg, ***@***.***> wrote: dies it load the library and modules? library(JuliaCall) julia_setup() Actually not. Opening R and calling those two commands crashes while doing julia_setup() The versions of my packages are identical to yours. A mistery ... Enclosed is a log of what I get .... strangely it seems to be using /usr/lib/R/lib/libR.so at the beginning even if later it uses the version from my environment /home/jost/miniconda3/envs/esmvaltool3/lib/R/bin/exec/../../lib/libR.so juliacall_crash.log <https://github.com/ESMValGroup/ESMValTool/files/3530997/juliacall_crash.log> In any case it may not be so important to investigate this further, since calling julia directly (instead of passing through R) should be preferrable, as implemented in #1242 <#1242> In #1240 <#1240> I also prepared an example template diagnostic in pure julia to be used for future development in julia. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1180?email_source=notifications&email_token=AG5EFI5QNZF4SX26CJWODCTQF23X3A5CNFSM4H4ENDN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD45TQOY#issuecomment-523974715>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFI7OIJ56BOADQ2MBRADQF23X3ANCNFSM4H4ENDNQ> .

[jhardenberg]

I managed to solve my problems with JuliaCall (an issue with my environment, related to julia using the wrong R when compiling the RCall package .... )
Apparently it is important to FIRST install R tools with Rscript esmvaltool/install/R/setup.R and THEN to install julia libraries with julia esmvaltool/install/Julia/setup.jl, in this order. I was doing the opposite and the julia RCall package would not compile correctly. I will add this info to the documentation.

In any case I can now confirm that using the #1254 julia installation method for me also the standard R-based rainfarm recipe works again, I do not encounter the problems documented by @valeriupredoi above. So we could close this issue.

As said above, if #1242 is adopted all this will become irrelevant anyway.

[mattiarighi]

This is also not working for me at DKRZ, where I'm installing julia following the standard instructions in CONTRIBUTING.md (i.e., not from conda).

The following error is returned:

esmvalcore._task.DiagnosticError: Diagnostic script rainfarm/rainfarm.R failed with return code -6. See the log in /mnt/lustre02/work/bd0854/b309057/ESMValTool/v2_output/recipe_rainfarm_20190905_143346/run/rainfarm/rainfarm/log.txt

This is the logfile from the diag script: log.txt.

[mattiarighi]

Sorry, I didn't notice there is already a PR on this. Testing now...