Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend #61 with tests and docs #74

Merged
merged 5 commits into from
Sep 24, 2020
Merged

Extend #61 with tests and docs #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f571f2a
Allow explicit environment specs
mathieu1 Aug 13, 2020
b291dfe
Use conda install for explicit environment specs
mathieu1 Aug 13, 2020
93b10d3
merging upstream
bollwyvl Sep 19, 2020
5c079ce
rebuild
bollwyvl Sep 19, 2020
9e65e8a
add test and docs
bollwyvl Sep 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/example-7.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: "Example 7: Explicit Environment Specification"

on:
pull_request:
branches:
- '*'
push:
branches:
- 'master'

jobs:
example-7:
name: Ex7 Explicit
runs-on: 'ubuntu-latest'
steps:
- uses: actions/checkout@v2
- uses: ./
with:
auto-update-conda: false
activate-environment: explicit-env
environment-file: etc/example-explicit.conda.lock
- shell: bash -l {0}
run: |
conda info
conda list
conda config --show-sources
conda config --show
printenv | sort
52 changes: 52 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,58 @@ jobs:
run: mamba install jupyterlab
```

### Example 7: Explicit Specification

`conda list --explicit` and [conda-lock][] support generating
[explicit environment specifications][spec], which skip the environment
solution step altogether, as they contain the _ordered_ list of exact URLs needed
to reproduce the environment.

This means explicitly-defined environments...

- are _much faster_ to install, as several expensive steps are skipped:
- channels are not queried for their repo data
- no solver is run
- are not cross-platform, as the URLs almost always contain platform/architecture information
- can become broken if any file becomes unavailable

This approach can be useful as part of a larger system e.g. a separate workflow that
runs `conda-lock` for all the platforms needed in a separate job.

[conda-lock]: https://github.com/conda-incubator/conda-lock
[spec]: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments

```yaml
name: "Example 7: Explicit Environment Specification"

on:
pull_request:
branches:
- '*'
push:
branches:
- 'master'

jobs:
example-7:
name: Ex7 Explicit
runs-on: 'ubuntu-latest'
steps:
- uses: actions/checkout@v2
- uses: conda-incubator/setup-miniconda@v1
with:
auto-update-conda: false
activate-environment: explicit-env
environment-file: etc/example-explicit.conda.lock
- shell: bash -l {0}
run: |
conda info
conda list
conda config --show-sources
conda config --show
printenv | sort
```

## Caching

If you want to enable package caching for conda you can use the [cache action](https://github.com/actions/cache) using `~/conda_pkgs_dir` as path for conda packages.
Expand Down
40 changes: 31 additions & 9 deletions dist/setup/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21875,15 +21875,27 @@ function setupMiniconda(installerUrl, minicondaVersion, architecture, condaVersi
}
// Read the environment yaml to use channels if provided and avoid conda solver conflicts
let environmentYaml;
let environmentExplicit;
if (environmentFile) {
try {
const sourceEnvironmentPath = path.join(process.env["GITHUB_WORKSPACE"] || "", environmentFile);
environmentYaml = yield yaml.safeLoad(fs.readFileSync(sourceEnvironmentPath, "utf8"));
environmentExplicit = fs
.readFileSync(sourceEnvironmentPath, "utf8")
.includes("\n@EXPLICIT\n");
if (environmentExplicit) {
environmentYaml = {};
}
else {
environmentYaml = yield yaml.safeLoad(fs.readFileSync(sourceEnvironmentPath, "utf8"));
}
}
catch (err) {
return { ok: false, error: err };
}
}
else {
environmentExplicit = false;
}
let cacheFolder = "~/conda_pkgs_dir";
cacheFolder = cacheFolder.replace("~", os.homedir().replace("\\", "/"));
result = yield condaCommand(`config --add pkgs_dirs ${cacheFolder}`, useBundled, useMamba);
Expand Down Expand Up @@ -21969,35 +21981,45 @@ function setupMiniconda(installerUrl, minicondaVersion, architecture, condaVersi
let activateEnvironmentToUse;
try {
const sourceEnvironmentPath = path.join(process.env["GITHUB_WORKSPACE"] || "", environmentFile);
environmentYaml = yield yaml.safeLoad(fs.readFileSync(sourceEnvironmentPath, "utf8"));
if (environmentExplicit) {
environmentYaml = {};
}
else {
environmentYaml = yield yaml.safeLoad(fs.readFileSync(sourceEnvironmentPath, "utf8"));
}
}
catch (err) {
return { ok: false, error: err };
}
if (activateEnvironment &&
if (environmentExplicit) {
condaAction = "install";
activateEnvironmentToUse = activateEnvironment;
utils.consoleLog(`Creating conda environment from explicit specs file...`);
}
else if (activateEnvironment &&
environmentYaml["name"] !== undefined &&
environmentYaml["name"] !== activateEnvironment) {
condaAction = "create";
condaAction = "env create";
activateEnvironmentToUse = environmentYaml["name"];
utils.consoleLog(`Creating conda environment from yaml file...`);
core.warning('The environment name on "environment-file" is not the same as "enviroment-activate", using "environment-file"!');
}
else if (activateEnvironment &&
activateEnvironment === environmentYaml["name"]) {
utils.consoleLog(`Updating conda environment from yaml file...`);
condaAction = "update";
condaAction = "env update";
activateEnvironmentToUse = activateEnvironment;
}
else if (activateEnvironment && environmentYaml["name"] === undefined) {
core.warning('The environment name on "environment-file" is not defined, using "enviroment-activate"!');
condaAction = "update";
condaAction = "env update";
activateEnvironmentToUse = activateEnvironment;
}
else {
activateEnvironmentToUse = activateEnvironment;
condaAction = "create";
condaAction = "env create";
}
result = yield condaCommand(`env ${condaAction} --file ${environmentFile} --name ${activateEnvironmentToUse}`, useBundled, useMamba);
result = yield condaCommand(`${condaAction} --file ${environmentFile} --name ${activateEnvironmentToUse}`, useBundled, useMamba);
if (!result.ok)
return result;
}
Expand Down Expand Up @@ -29134,7 +29156,7 @@ module.exports = defaults;
/* 771 */
/***/ (function(module) {

module.exports = {"_from":"cheerio@^1.0.0-rc.2","_id":"[email protected]","_inBundle":false,"_integrity":"sha512-0td5ijfUPuubwLUu0OBoe98gZj8C/AA+RW3v67GPlGOrvxWjZmBXiBCRU+I8VEiNyJzjth40POfHiz2RB3gImA==","_location":"/cheerio","_phantomChildren":{},"_requested":{"type":"range","registry":true,"raw":"cheerio@^1.0.0-rc.2","name":"cheerio","escapedName":"cheerio","rawSpec":"^1.0.0-rc.2","saveSpec":null,"fetchSpec":"^1.0.0-rc.2"},"_requiredBy":["/get-hrefs"],"_resolved":"https://registry.npmjs.org/cheerio/-/cheerio-1.0.0-rc.3.tgz","_shasum":"094636d425b2e9c0f4eb91a46c05630c9a1a8bf6","_spec":"cheerio@^1.0.0-rc.2","_where":"/Users/goanpeca/Dropbox (Personal)/develop/quansight/setup-miniconda/node_modules/get-hrefs","author":{"name":"Matt Mueller","email":"[email protected]","url":"mat.io"},"bugs":{"url":"https://github.com/cheeriojs/cheerio/issues"},"bundleDependencies":false,"dependencies":{"css-select":"~1.2.0","dom-serializer":"~0.1.1","entities":"~1.1.1","htmlparser2":"^3.9.1","lodash":"^4.15.0","parse5":"^3.0.1"},"deprecated":false,"description":"Tiny, fast, and elegant implementation of core jQuery designed specifically for the server","devDependencies":{"benchmark":"^2.1.0","coveralls":"^2.11.9","expect.js":"~0.3.1","istanbul":"^0.4.3","jquery":"^3.0.0","jsdom":"^9.2.1","jshint":"^2.9.2","mocha":"^3.1.2","xyz":"~1.1.0"},"engines":{"node":">= 0.6"},"files":["index.js","lib"],"homepage":"https://github.com/cheeriojs/cheerio#readme","keywords":["htmlparser","jquery","selector","scraper","parser","html"],"license":"MIT","main":"./index.js","name":"cheerio","repository":{"type":"git","url":"git://github.com/cheeriojs/cheerio.git"},"scripts":{"test":"make test"},"version":"1.0.0-rc.3"};
module.exports = {"_args":[["[email protected]","/home/weg/projects/actions/setup-miniconda"]],"_from":"[email protected]","_id":"[email protected]","_inBundle":false,"_integrity":"sha512-0td5ijfUPuubwLUu0OBoe98gZj8C/AA+RW3v67GPlGOrvxWjZmBXiBCRU+I8VEiNyJzjth40POfHiz2RB3gImA==","_location":"/cheerio","_phantomChildren":{},"_requested":{"type":"version","registry":true,"raw":"[email protected]","name":"cheerio","escapedName":"cheerio","rawSpec":"1.0.0-rc.3","saveSpec":null,"fetchSpec":"1.0.0-rc.3"},"_requiredBy":["/get-hrefs"],"_resolved":"https://registry.npmjs.org/cheerio/-/cheerio-1.0.0-rc.3.tgz","_spec":"1.0.0-rc.3","_where":"/home/weg/projects/actions/setup-miniconda","author":{"name":"Matt Mueller","email":"[email protected]","url":"mat.io"},"bugs":{"url":"https://github.com/cheeriojs/cheerio/issues"},"dependencies":{"css-select":"~1.2.0","dom-serializer":"~0.1.1","entities":"~1.1.1","htmlparser2":"^3.9.1","lodash":"^4.15.0","parse5":"^3.0.1"},"description":"Tiny, fast, and elegant implementation of core jQuery designed specifically for the server","devDependencies":{"benchmark":"^2.1.0","coveralls":"^2.11.9","expect.js":"~0.3.1","istanbul":"^0.4.3","jquery":"^3.0.0","jsdom":"^9.2.1","jshint":"^2.9.2","mocha":"^3.1.2","xyz":"~1.1.0"},"engines":{"node":">= 0.6"},"files":["index.js","lib"],"homepage":"https://github.com/cheeriojs/cheerio#readme","keywords":["htmlparser","jquery","selector","scraper","parser","html"],"license":"MIT","main":"./index.js","name":"cheerio","repository":{"type":"git","url":"git://github.com/cheeriojs/cheerio.git"},"scripts":{"test":"make test"},"version":"1.0.0-rc.3"};

/***/ }),
/* 772 */
Expand Down
57 changes: 57 additions & 0 deletions etc/example-explicit.conda.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# platform: linux-64
# env_hash: d6eb453f6a468e35232d208ad5bf41236beba34621f68412ce983279272075e0
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.conda#c3473ff8bdb3d124ed5ff11ec380d6f9
https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2020.7.22-0.conda#3efde69d1bd20bac0b3be953e0c26114
https://repo.anaconda.com/pkgs/main/linux-64/ld_impl_linux-64-2.33.1-h53a641e_7.conda#ae7fa9dd2ad15b81ab2ba25fe264bcb0
https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.conda#c3efd14430bedbe0a036a899f50cec7f
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.conda#80a23105874059534307086faae566c5
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.3-he6710b0_2.conda#88a54b8f50e351c650e16f4ee781440c
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.2-he6710b0_1.conda#b7ddf5ee3934b7f54620e8f6c0ee8a95
https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1g-h7b6447c_0.conda#d64818ba115b875b04e1cc23851c0851
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.5-h7b6447c_0.conda#73a16ea8ba890175a1ce94a3a87c8f68
https://repo.anaconda.com/pkgs/main/linux-64/yaml-0.2.5-h7b6447c_0.conda#39fdbf4db769e494ffb06d95680c83d8
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.conda#50aa4407424985d56ec8ae2f94626aa9
https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20191231-h14c3975_1.conda#a900fd130232e0189f071fabbc6025d9
https://repo.anaconda.com/pkgs/main/linux-64/readline-8.0-h7b6447c_0.conda#a0872d6b7b037b1e073c962140ae72aa
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.10-hbc83047_0.conda#9ba14aaba4818a66c820f85f5bf34ca0
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.33.0-h62c20be_0.conda#c4d7dd5cb9eca27fdf96764c1f706cb5
https://repo.anaconda.com/pkgs/main/linux-64/python-3.8.5-h7579374_1.conda#fa19bb6876427f0aadd3e2c2804fbb61
https://repo.anaconda.com/pkgs/main/noarch/appdirs-1.4.4-py_0.conda#483878b172607008f5f8e3bfded615bc
https://repo.anaconda.com/pkgs/main/noarch/attrs-20.2.0-py_0.conda#c6d3c37f35da39570348537d6abfcf94
https://repo.anaconda.com/pkgs/main/linux-64/certifi-2020.6.20-py38_0.conda#7881da1725fe5021071979e3566e3141
https://repo.anaconda.com/pkgs/main/linux-64/chardet-3.0.4-py38_1003.conda#c28b6ec71e01bbe6200d6953f67e4ef4
https://repo.anaconda.com/pkgs/main/noarch/click-7.1.2-py_0.conda#122e0ead7703cf4f04a36b34e2d1f9a2
https://repo.anaconda.com/pkgs/main/noarch/decorator-4.4.2-py_0.conda#3f5146ff080437ecb0e1583a6021c634
https://repo.anaconda.com/pkgs/main/noarch/idna-2.10-py_0.conda#72085bc43b003e02a3ff9356bc428f25
https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py38_0.conda#82996fcc5e928d5e50535d16c7eb7b08
https://repo.anaconda.com/pkgs/main/linux-64/mypy_extensions-0.4.3-py38_0.conda#f6e530e21a804080c85ba3c45ba33e0d
https://repo.anaconda.com/pkgs/main/noarch/pathspec-0.7.0-py_0.conda#948c37b334786989007133a6e07800bc
https://repo.anaconda.com/pkgs/main/noarch/pycparser-2.20-py_2.conda#fcfeb621c6f895f3562ff01d9d6ce959
https://repo.anaconda.com/pkgs/main/linux-64/pyrsistent-0.17.3-py38h7b6447c_0.conda#ba41d756b401b9e76f94d8d44526ec9a
https://repo.anaconda.com/pkgs/main/linux-64/pysocks-1.7.1-py38_0.conda#148079353d446b5a99b5082682b975af
https://repo.anaconda.com/pkgs/main/noarch/pytz-2020.1-py_0.conda#01cec12ecd4a6e162d4ed2349929e81b
https://repo.anaconda.com/pkgs/main/linux-64/pyyaml-5.3.1-py38h7b6447c_1.conda#13572270cd3f42590968ce4885dc2e04
https://repo.anaconda.com/pkgs/main/linux-64/regex-2020.7.14-py38h7b6447c_0.conda#30d596eecf2fd0d92326e82fe201b808
https://repo.anaconda.com/pkgs/main/noarch/six-1.15.0-py_0.conda#54c2e03874b7bba2404be8ee23148927
https://repo.anaconda.com/pkgs/main/noarch/toml-0.10.1-py_0.conda#1b644e2153dff00c27514b97d74fbd30
https://repo.anaconda.com/pkgs/main/linux-64/typed-ast-1.4.1-py38h7b6447c_0.conda#293e50c79187f9bbc326693465f7ed60
https://repo.anaconda.com/pkgs/main/noarch/typing_extensions-3.7.4.3-py_0.conda#c26af7e476ebc6e70a6005c4492eb18d
https://repo.anaconda.com/pkgs/main/noarch/zipp-3.1.0-py_0.conda#1f4b6d7b01b2e3a8c480454af14c7da8
https://repo.anaconda.com/pkgs/main/noarch/black-19.10b0-py_0.conda#dda9c1964ca705ed9dc055eb3a3d813a
https://repo.anaconda.com/pkgs/main/linux-64/cffi-1.14.2-py38he30daa8_0.conda#88a65218813428797b75c6613c5a88a6
https://repo.anaconda.com/pkgs/main/linux-64/importlib-metadata-1.7.0-py38_0.conda#f09caad8b5d640f732db3b96d98457ff
https://repo.anaconda.com/pkgs/main/noarch/python-dateutil-2.8.1-py_0.conda#062ee583dae450d908d45d6c08beae97
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-49.6.0-py38_0.conda#63e562de6af837707e866516aa275b89
https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.3-py38_0.conda#25565a2af22674ec46100dcd3d675d1f
https://repo.anaconda.com/pkgs/main/linux-64/brotlipy-0.7.0-py38h7b6447c_1000.conda#cb72879bd0a40e155cd4a5bc02b0dcdb
https://repo.anaconda.com/pkgs/main/linux-64/clyent-1.2.2-py38_1.conda#178f9f5f690b893d3bb9d450db581adc
https://repo.anaconda.com/pkgs/main/linux-64/cryptography-3.1-py38h1ba5d50_0.conda#3d2c0ee3870267cbd4632a2cc727f583
https://repo.anaconda.com/pkgs/main/noarch/importlib_metadata-1.7.0-0.conda#23f2c6fd9b40edc06aea0623a122fe1e
https://repo.anaconda.com/pkgs/main/linux-64/jupyter_core-4.6.3-py38_0.conda#369fa0f64f4f1ac29ecebf094be83989
https://repo.anaconda.com/pkgs/main/linux-64/jsonschema-3.2.0-py38_0.conda#8c491f8f33feae1aee5b17a4caa0a751
https://repo.anaconda.com/pkgs/main/linux-64/pyopenssl-19.1.0-py38_0.conda#c6d2e2b7f5bdb5516e92e4da5e0b4fbe
https://repo.anaconda.com/pkgs/main/noarch/nbformat-5.0.7-py_0.conda#a79a6b8519be83d6fcc541b73d6b9135
https://repo.anaconda.com/pkgs/main/noarch/urllib3-1.25.10-py_0.conda#cd34894b40e348faeac50f72994c6f6b
https://repo.anaconda.com/pkgs/main/noarch/requests-2.24.0-py_0.conda#7e39484a07f30dd32a52bed7306bff02
https://repo.anaconda.com/pkgs/main/linux-64/anaconda-client-1.7.2-py38_0.conda#ddc0ff43c2836ac0b59042010ed08592
2 changes: 1 addition & 1 deletion package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 32 additions & 12 deletions src/setup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,18 +707,28 @@ async function setupMiniconda(

// Read the environment yaml to use channels if provided and avoid conda solver conflicts
let environmentYaml: any;
let environmentExplicit: boolean;
if (environmentFile) {
try {
const sourceEnvironmentPath: string = path.join(
process.env["GITHUB_WORKSPACE"] || "",
environmentFile
);
environmentYaml = await yaml.safeLoad(
fs.readFileSync(sourceEnvironmentPath, "utf8")
);
environmentExplicit = fs
.readFileSync(sourceEnvironmentPath, "utf8")
.includes("\n@EXPLICIT\n");
if (environmentExplicit) {
environmentYaml = {};
} else {
environmentYaml = await yaml.safeLoad(
fs.readFileSync(sourceEnvironmentPath, "utf8")
);
}
} catch (err) {
return { ok: false, error: err };
}
} else {
environmentExplicit = false;
}

let cacheFolder: string = "~/conda_pkgs_dir";
Expand Down Expand Up @@ -850,18 +860,28 @@ async function setupMiniconda(
process.env["GITHUB_WORKSPACE"] || "",
environmentFile
);
environmentYaml = await yaml.safeLoad(
fs.readFileSync(sourceEnvironmentPath, "utf8")
);
if (environmentExplicit) {
environmentYaml = {};
} else {
environmentYaml = await yaml.safeLoad(
fs.readFileSync(sourceEnvironmentPath, "utf8")
);
}
} catch (err) {
return { ok: false, error: err };
}
if (
if (environmentExplicit) {
condaAction = "install";
activateEnvironmentToUse = activateEnvironment;
utils.consoleLog(
`Creating conda environment from explicit specs file...`
);
} else if (
activateEnvironment &&
environmentYaml["name"] !== undefined &&
environmentYaml["name"] !== activateEnvironment
) {
condaAction = "create";
condaAction = "env create";
activateEnvironmentToUse = environmentYaml["name"];
utils.consoleLog(`Creating conda environment from yaml file...`);
core.warning(
Expand All @@ -872,20 +892,20 @@ async function setupMiniconda(
activateEnvironment === environmentYaml["name"]
) {
utils.consoleLog(`Updating conda environment from yaml file...`);
condaAction = "update";
condaAction = "env update";
activateEnvironmentToUse = activateEnvironment;
} else if (activateEnvironment && environmentYaml["name"] === undefined) {
core.warning(
'The environment name on "environment-file" is not defined, using "enviroment-activate"!'
);
condaAction = "update";
condaAction = "env update";
activateEnvironmentToUse = activateEnvironment;
} else {
activateEnvironmentToUse = activateEnvironment;
condaAction = "create";
condaAction = "env create";
}
result = await condaCommand(
`env ${condaAction} --file ${environmentFile} --name ${activateEnvironmentToUse}`,
`${condaAction} --file ${environmentFile} --name ${activateEnvironmentToUse}`,
useBundled,
useMamba
);
Expand Down