Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce CONDA_JL_CONDA_EXE environment variable #216

Merged
merged 12 commits into from
Feb 8, 2022
Merged

Introduce CONDA_JL_CONDA_EXE environment variable #216

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3d42c9f
Introduce CONDA_JL_CONDA_EXE environment variable
mkitti Jan 15, 2022
53ed58c
Error and recommend rebuild if CONDA_EXE does not exist
mkitti Jan 15, 2022
6c244f2
Add Conda.create, use for testing
mkitti Jan 15, 2022
1399e71
Change default CONDA_EXE to match ROOTENV if version mismatch
mkitti Jan 15, 2022
bbb2653
Fix typos in tests
mkitti Jan 15, 2022
2162971
Use local variables instead of let
mkitti Jan 15, 2022
0756e35
The folder is Scripts not Script, fix Windows errors
mkitti Jan 15, 2022
c95e1ab
Check if CONDA_EXE exists CONDA_JL_CONDA_EXE is set during build
mkitti Jan 15, 2022
8eb7c9a
Check if CONDA_JL_CONDA_EXE exists & executable. Install only in ROOTENV
mkitti Jan 15, 2022
1cadc27
Update test/runtests.jl
tkf Jan 16, 2022
2c723b7
Add doc for `CONDA_JL_CONDA_EXE` to README.md
mkitti Jan 16, 2022
84c662c
Use Sys.isexecutable in deps.jl
mkitti Jan 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name = "Conda"
uuid = "8f4d0f93-b110-5947-807f-2305c1781a2d"
version = "1.6"
version = "1.7"

[deps]
Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
Expand Down
42 changes: 42 additions & 0 deletions deps/build.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,19 @@ Using the Anaconda/defaults channel instead, which is free for non-commercial us
if !isdefined(@__MODULE__, :USE_MINIFORGE)
const USE_MINIFORGE = USE_MINIFORGE_DEFAULT
end
function default_conda_exe(ROOTENV)
@static if Sys.iswindows()
p = joinpath(ROOTENV, "Scripts")
conda_bat = joinpath(p, "conda.bat")
isfile(conda_bat) ? conda_bat : joinpath(p, "conda.exe")
else
joinpath(ROOTENV, "bin", "conda")
end
end

if !isdefined(@__MODULE__, :CONDA_EXE)
const CONDA_EXE = default_conda_exe(ROOTENV)
end
end

MINICONDA_VERSION = get(ENV, "CONDA_JL_VERSION", DefaultDeps.MINICONDA_VERSION)
Expand Down Expand Up @@ -52,10 +65,39 @@ These will require rebuilding.
""")
end

CONDA_EXE = get(ENV, "CONDA_JL_CONDA_EXE") do
if ROOTENV == DefaultDeps.ROOTENV
DefaultDeps.CONDA_EXE
else
DefaultDeps.default_conda_exe(ROOTENV)
end
end

if haskey(ENV, "CONDA_JL_CONDA_EXE")
# Check to see if CONDA_EXE is an executable file
if isfile(CONDA_EXE)
if uperm(CONDA_EXE) & 0x01 > 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sys.isexecutable(CONDA_EXE)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I made this change in 84c662c

@info "Executable conda located." CONDA_EXE
else
error("CONDA_JL_CONDA_EXE, $CONDA_EXE, cannot be executed by the current user.")
end
else
error("CONDA_JL_CONDA_EXE, $CONDA_EXE, does not exist.")
end
else
if !isfile(CONDA_EXE)
# An old CONDA_EXE has gone missing, revert to default in ROOTENV
@info "CONDA_EXE not found. Reverting to default in ROOTENV" CONDA_EXE ROOTENV
CONDA_EXE = DefaultDeps.default_conda_exe(ROOTENV)
end
end


deps = """
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "$(escape_string(MINICONDA_VERSION))"
const USE_MINIFORGE = $USE_MINIFORGE
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""

mkpath(condadir)
Expand Down
19 changes: 11 additions & 8 deletions src/Conda.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,14 +70,12 @@ function python_dir(env::Environment)
end
const PYTHONDIR = python_dir(ROOTENV)

# note: the same conda program is used for all environments
const conda = if Sys.iswindows()
p = script_dir(ROOTENV)
conda_bat = joinpath(p, "conda.bat")
isfile(conda_bat) ? conda_bat : joinpath(p, "conda.exe")
else
joinpath(bin_dir(ROOTENV), "conda")
if ! @isdefined(CONDA_EXE)
# We have an oudated deps.jl file that does not define CONDA_EXE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is impossible. If there's an outdated deps.jl that means there was a update of Conda.jl which means the package has to be built first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible in a few situations such as using multiple Julia installations with the same JULIA_DEPOT_PATH and/or a modified JULIA_LOAD_PATH. Julia will only force a rebuild if it is aware that the version changed.

For example, I did this to myself in the course of testing the package by using pkg"dev Conda". In this case, the Conda package was checked out in ~/.julia/dev/Conda and the deps file at ~/.julia/dev/Conda/deps/deps.jlwas carried over between several versions as I moved between branches or I changed environments.

While we could just throw and error if we detect this condition, encouraging the user to rebuild, just doing the former calculation seems friendlier since we can just follow the old behavior here.

We see this issue with an outdated deps.jl file quite commonly with GR.jl and users will appear in support channels complaining that plotting is broken. The issues were reduced once we just started fixing the issues automatically.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove it now, but my recommendation is to keep it for the moment and then remove it at a later time to smooth over the migration process.

error("CONDA_EXE not defined in $deps_file.\nPlease rebuild Conda.jl via `using Pkg; pkg\"build Conda\";`")
end
# note: the same conda program is used for all environments
const conda = CONDA_EXE

"Path to the condarc file"
function conda_rc(env::Environment)
Expand Down Expand Up @@ -191,6 +189,7 @@ _quiet() = get(ENV, "CI", "false") == "true" ? `-q` : ``
"Install miniconda if it hasn't been installed yet; _install_conda(true) installs Conda even if it has already been installed."
function _install_conda(env::Environment, force::Bool=false)
if force || !isfile(Conda.conda)
@assert startswith(abspath(Conda.conda), abspath(PREFIX)) "CONDA_EXE, $(conda), does not exist within $PREFIX"
@info("Downloading miniconda installer ...")
if Sys.isunix()
installer = joinpath(PREFIX, "installer.sh")
Expand All @@ -211,7 +210,7 @@ function _install_conda(env::Environment, force::Bool=false)
end
end
if !isdir(prefix(env))
runconda(`create $(_quiet()) -y -p $(prefix(env))`)
create(env)
end
end

Expand All @@ -228,6 +227,10 @@ function rm(pkg::PkgOrPkgs, env::Environment=ROOTENV)
runconda(`remove $(_quiet()) -y $pkg`, env)
end

function create(env::Environment)
runconda(`create $(_quiet()) -y -p $(prefix(env))`)
end

"Update all installed packages."
function update(env::Environment=ROOTENV)
if env == ROOTENV
Expand Down
107 changes: 72 additions & 35 deletions test/runtests.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,36 +70,42 @@ Conda.rm_channel("foo", env)
Conda.add("zlib", env; channel=alt_channel)

@testset "Batch install and uninstall" begin
Conda.add(["affine", "ansi2html"], env)
installed = Conda._installed_packages(env)
@test "affine" ∈ installed
@test "ansi2html" ∈ installed

Conda.rm(["affine", "ansi2html"], env)
installed = Conda._installed_packages(env)
@test "affine" ∉ installed
@test "ansi2html" ∉ installed
mktempdir() do env
Conda.create(env)
Conda.add(["affine", "ansi2html"], env)
installed = Conda._installed_packages(env)
@test "affine" ∈ installed
@test "ansi2html" ∈ installed

Conda.rm(["affine", "ansi2html"], env)
installed = Conda._installed_packages(env)
@test "affine" ∉ installed
@test "ansi2html" ∉ installed
end
end

# Run conda clean
Conda.clean(; debug=true)

@testset "Exporting and creating environments" begin
new_env = :test_conda_jl_2
Conda.add("curl", env)
Conda.export_list("conda-pkg.txt", env)

# Create a new environment
rm(Conda.prefix(new_env); force=true, recursive=true)
Conda.import_list(
IOBuffer(read("conda-pkg.txt")), new_env; channels=["foo", alt_channel, default_channel]
)

# Ensure that our new environment has our channels and package installed.
@test Conda.channels(new_env) == ["foo", alt_channel, default_channel]
installed = Conda._installed_packages(new_env)
@test "curl" ∈ installed
rm("conda-pkg.txt")
mktempdir() do env
new_env = :test_conda_jl_2
Conda.create(env)
Conda.add("curl", env)
Conda.export_list("conda-pkg.txt", env)

# Create a new environment
rm(Conda.prefix(new_env); force=true, recursive=true)
Conda.import_list(
IOBuffer(read("conda-pkg.txt")), new_env; channels=["foo", alt_channel, default_channel]
)

# Ensure that our new environment has our channels and package installed.
@test Conda.channels(new_env) == ["foo", alt_channel, default_channel]
installed = Conda._installed_packages(new_env)
@test "curl" ∈ installed
rm("conda-pkg.txt")
end
end

@testset "Conda.pip_interop" begin
Expand Down Expand Up @@ -150,6 +156,17 @@ end
end
end

function default_conda_exe(ROOTENV)
@static if Sys.iswindows()
p = joinpath(ROOTENV, "Scripts")
conda_bat = joinpath(p, "conda.bat")
isfile(conda_bat) ? conda_bat : joinpath(p, "conda.exe")
else
joinpath(ROOTENV, "bin", "conda")
end
end


if Sys.ARCH in [:x86, :i686]
CONDA_JL_USE_MINIFORGE_DEFAULT = "false"
else
Expand All @@ -162,12 +179,15 @@ end
@test !isfile(depsfile)
@test !isfile(joinpath(condadir, "deps.jl"))

withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing) do
withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing, "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local ROOTENV=joinpath(condadir, "3")
local CONDA_EXE=default_conda_exe(ROOTENV)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(joinpath(condadir, "3")))"
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = $(CONDA_JL_USE_MINIFORGE_DEFAULT)
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end
end
Expand All @@ -179,12 +199,15 @@ end
@test !isfile(depsfile)
@test !isfile(joinpath(condadir, "deps.jl"))

withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => "1") do
withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => "1", "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local ROOTENV=joinpath(condadir, "3")
local CONDA_EXE=default_conda_exe(ROOTENV)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(joinpath(condadir, "3")))"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = true
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end
end
Expand All @@ -193,12 +216,15 @@ end
@test !isfile(depsfile)
@test !isfile(joinpath(condadir, "deps.jl"))

withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => "0") do
withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => "0", "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local ROOTENV=joinpath(condadir, "3")
local CONDA_EXE=default_conda_exe(ROOTENV)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(joinpath(condadir, "3")))"
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = false
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end
end
Expand All @@ -207,12 +233,14 @@ end
@testset "custom home" begin
preserve_build() do
mktempdir() do dir
withenv("CONDA_JL_VERSION" => "3", "CONDA_JL_HOME" => dir, "CONDA_JL_USE_MINIFORGE" => nothing) do
withenv("CONDA_JL_VERSION" => "3", "CONDA_JL_HOME" => dir, "CONDA_JL_USE_MINIFORGE" => nothing, "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local CONDA_EXE=default_conda_exe(dir)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(dir))"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = $(CONDA_JL_USE_MINIFORGE_DEFAULT)
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end
end
Expand All @@ -222,28 +250,37 @@ end
@testset "version mismatch" begin
preserve_build() do
# Mismatch in written file
local ROOTENV=joinpath(condadir, "3")
local CONDA_EXE=default_conda_exe(ROOTENV)
write(depsfile, """
const ROOTENV = "$(escape_string(joinpath(condadir, "3")))"
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "2"
const USE_MINIFORGE = $(CONDA_JL_USE_MINIFORGE_DEFAULT)
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
""")

withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing) do
withenv("CONDA_JL_VERSION" => nothing, "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing, "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local ROOTENV=joinpath(condadir, "2")
local CONDA_EXE=default_conda_exe(ROOTENV)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(joinpath(condadir, "2")))"
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "2"
const USE_MINIFORGE = $(CONDA_JL_USE_MINIFORGE_DEFAULT)
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end

# ROOTENV should be replaced since CONDA_JL_HOME wasn't explicitly set
withenv("CONDA_JL_VERSION" => "3", "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing) do
withenv("CONDA_JL_VERSION" => "3", "CONDA_JL_HOME" => nothing, "CONDA_JL_USE_MINIFORGE" => nothing, "CONDA_JL_CONDA_EXE" => nothing) do
Pkg.build("Conda")
local ROOTENV=joinpath(condadir, "3")
local CONDA_EXE=default_conda_exe(ROOTENV)
@test read(depsfile, String) == """
const ROOTENV = "$(escape_string(joinpath(condadir, "3")))"
const ROOTENV = "$(escape_string(ROOTENV))"
const MINICONDA_VERSION = "3"
const USE_MINIFORGE = $(CONDA_JL_USE_MINIFORGE_DEFAULT)
const CONDA_EXE = "$(escape_string(CONDA_EXE))"
"""
end
end
Expand Down