From 990f3e6ab093a5727f193b31dbec58f29799db87 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 4 Jun 2024 11:11:52 +0200 Subject: [PATCH 1/5] Standardize `environment.yml` --- cep-??.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 cep-??.md diff --git a/cep-??.md b/cep-??.md new file mode 100644 index 00000000..4eb0a185 --- /dev/null +++ b/cep-??.md @@ -0,0 +1,132 @@ + + + + + + + + +
Title Specification of `environment.yml` input files
Status Draft
Author(s) Jaime Rodríguez-Guerra <jaime.rogue@gmail.com>
Created June 4, 2024
Updated June 4, 2024
Discussion https://github.com/conda/ceps/pull/80
Implementation NA
+ +## Abstract + +This CEP standardizes the format of `environment.yml` input files for conda clients. + +## Motivation + +The motivation of this CEP is merely informative. It describes the details of an existing file format. + +## Nomenclature + +The `environment.yml` file format was introduced by `conda env`. It has been traditionally referred to as "the YAML input file", but other YAML files exist in the ecosystem now. To avoid confusion, in this CEP we will only use the term `environment.yml`. Do note that the actual filename of an input file using this format might be a different one (e.g. `env.yml`). + +## Specification + +`environment.yml` files are YAML documents that encode the necessary information to create a new conda environment. They can contain up to five top-level keys: + +- `name`: The preferred name for the environment. +- `prefix`: The preferred full path for the environment. Often ignored. +- `dependencies`: List of package specifications that MUST be installed in the new environment. +- `channels`: conda channels that will be used to solve the dependencies +- `variables`: Environment variables that SHOULD be added to the `conda-meta/state` file in the resulting environment. + +Additional sections can be present, but they MUST be ignored and the user SHOULD receive an informative warning about them. + +The file name is not standardized, but the extension MUST be `yml` or `yaml`. + +### `name` and `prefix` + +Optional, `str`. + +Both fields refer to the _preferred_ name or path for the newly created environment. Tools SHOULD allow these suggestions to be overridden by the user with additional CLI flags or equivalent. If the proposed environment path exists, tools MUST NOT overwrite silently by default. + +Special names `base` and `root` SHOULD not be accepted. Prefixes targetting protected system paths SHOULD be rejected. Paths can contain tildes (`~`) and environment variables, and they MUST be expanded when present. + +The name of the environment (and the last component of the expanded`prefix` path) MUST NOT contain any of these characters: `/`, ` `, `:`, `#`. + +### `dependencies` + +Required, `list[str | dict[str, Any]]`. + +The simplest form for this section is a list of `str` encoding `MatchSpec`-compatible requirements. These items MUST be processed as conda requirements and submitted to the solver (along with `channels`) to obtain the contents for the resulting conda environment. + +This section can also contain "subsection" dictionaries that map `str` to arbitrary values. Each key refers to a non-conda installer tool that will process the associated contents as necessary. The additional subsections MUST be processed after the conda requirements. They SHOULD only add new contents. These keys SHOULD NOT overwrite existing contents. + +Currently known subsections include `pip`, the contents of which encode a list of `str` referring to PyPI requirements. How to process this list is left as an implementation detail, but common approaches involve invoking the `pip` command-line directly. + +Additional subsections are allowed. The conda client MUST error out if it cannot process unknown sections. + +### `channels` + +Optional, `list[str]`. + +These are the conda channels that will be queried to solve the requirements added in `dependencies`. They can be expressed with names, URLs and local paths. + +If not specified, the conda client MUST use the default configuration. When specified, these channels MUST be used to populate the channel list passed to the solver. Then default channel configuration MUST be appended, unless the special name `nodefaults` is present. When this is the case, the default configuration MUST no be appended. The `nodefaults` name MUST not be passed to the solver. + +### `variables` + +Optional, `dict[str, str]`. + +Keys MUST be valid environment variable names for the target operating system. We recommend sticking to names compatible with both Windows and POSIX standards. Values SHOULD be `str`. If they are not, they MUST be converted to `str`. + +These contents SHOULD be dumped into the `conda-meta/state` file of the target environment, or equivalent, so they can be set upon environment activation. + +## Examples + +Simplest possible `environment.yml`: + +```yaml +dependencies: + - numpy +``` + +With a name: + +```yaml +name: test +dependencies: + - numpy >=1.10 +``` + +With channels: + +```yaml +name: test +channels: + - conda-forge +dependencies: + - numpy +``` + +With a `pip` section: + +```yaml +name: test +channels: + - conda-forge +dependencies: + - numpy + - pip: + - scipy +``` + +With variables: + +```yaml +name: test +channels: + - conda-forge +dependencies: + - numpy +variables: + MY_ENV_VAR: "My Value" +``` + +## Reference + +... + +## Copyright + +All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). From b7ebb7bd7d3a3c954cb834e713ed0b7fe857894b Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 4 Jun 2024 11:13:06 +0200 Subject: [PATCH 2/5] Update PR number --- cep-??.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-??.md b/cep-??.md index 4eb0a185..eb3531bc 100644 --- a/cep-??.md +++ b/cep-??.md @@ -4,7 +4,7 @@ Author(s) Jaime Rodríguez-Guerra <jaime.rogue@gmail.com> Created June 4, 2024 Updated June 4, 2024 - Discussion https://github.com/conda/ceps/pull/80 + Discussion https://github.com/conda/ceps/pull/81 Implementation NA From 7f47c78d06ab2f7ced67d4c9be0c1c78553e9b6b Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 4 Jun 2024 11:18:02 +0200 Subject: [PATCH 3/5] add references --- cep-??.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cep-??.md b/cep-??.md index eb3531bc..35a20714 100644 --- a/cep-??.md +++ b/cep-??.md @@ -125,7 +125,9 @@ variables: ## Reference -... +- [Allowed extensions](https://github.com/conda/conda/blob/24.5.0/conda/env/specs/yaml_file.py) +- [Allowed keys](https://github.com/conda/conda/blob/24.5.0/conda/env/env.py#L25) +- [Extra subsections in `dependencies`](https://github.com/conda/conda/blob/841d9d57fd96ad27cda4b7c43549104a96f961ce/conda/cli/main_env_create.py#L167-L185) ## Copyright From cd38671dc6eb8d54643b3d0192a34f5bdbc705c5 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 4 Jun 2024 11:20:00 +0200 Subject: [PATCH 4/5] Formatting --- cep-??.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cep-??.md b/cep-??.md index 35a20714..5245d83f 100644 --- a/cep-??.md +++ b/cep-??.md @@ -1,5 +1,5 @@ - + @@ -26,7 +26,7 @@ The `environment.yml` file format was introduced by `conda env`. It has been tra - `name`: The preferred name for the environment. - `prefix`: The preferred full path for the environment. Often ignored. -- `dependencies`: List of package specifications that MUST be installed in the new environment. +- `dependencies`: List of package specifications that MUST be installed in the new environment. - `channels`: conda channels that will be used to solve the dependencies - `variables`: Environment variables that SHOULD be added to the `conda-meta/state` file in the resulting environment. @@ -36,9 +36,9 @@ The file name is not standardized, but the extension MUST be `yml` or `yaml`. ### `name` and `prefix` -Optional, `str`. +Optional, `str`. -Both fields refer to the _preferred_ name or path for the newly created environment. Tools SHOULD allow these suggestions to be overridden by the user with additional CLI flags or equivalent. If the proposed environment path exists, tools MUST NOT overwrite silently by default. +Both fields refer to the _preferred_ name or path for the newly created environment. Tools SHOULD allow these suggestions to be overridden by the user with additional CLI flags or equivalent. If the proposed environment path exists, tools MUST NOT overwrite silently by default. Special names `base` and `root` SHOULD not be accepted. Prefixes targetting protected system paths SHOULD be rejected. Paths can contain tildes (`~`) and environment variables, and they MUST be expanded when present. From 2ae140c3c8a6f4b8bc14dc2f39b0c1b94f8ab038 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 5 Jun 2024 09:33:46 +0200 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Marco Esters --- cep-??.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cep-??.md b/cep-??.md index 5245d83f..9c8959c5 100644 --- a/cep-??.md +++ b/cep-??.md @@ -25,9 +25,9 @@ The `environment.yml` file format was introduced by `conda env`. It has been tra `environment.yml` files are YAML documents that encode the necessary information to create a new conda environment. They can contain up to five top-level keys: - `name`: The preferred name for the environment. -- `prefix`: The preferred full path for the environment. Often ignored. +- `prefix`: The preferred full path to the environment. Often ignored. - `dependencies`: List of package specifications that MUST be installed in the new environment. -- `channels`: conda channels that will be used to solve the dependencies +- `channels`: conda channels that will be used to resolve the dependencies. - `variables`: Environment variables that SHOULD be added to the `conda-meta/state` file in the resulting environment. Additional sections can be present, but they MUST be ignored and the user SHOULD receive an informative warning about them. @@ -40,9 +40,9 @@ Optional, `str`. Both fields refer to the _preferred_ name or path for the newly created environment. Tools SHOULD allow these suggestions to be overridden by the user with additional CLI flags or equivalent. If the proposed environment path exists, tools MUST NOT overwrite silently by default. -Special names `base` and `root` SHOULD not be accepted. Prefixes targetting protected system paths SHOULD be rejected. Paths can contain tildes (`~`) and environment variables, and they MUST be expanded when present. +Special names `base` and `root` SHOULD NOT be accepted. Prefixes targetting protected system paths SHOULD be rejected. Paths can contain tildes (`~`) and environment variables, and they MUST be expanded when present. -The name of the environment (and the last component of the expanded`prefix` path) MUST NOT contain any of these characters: `/`, ` `, `:`, `#`. +The name of the environment (and the last component of the expanded `prefix` path) MUST NOT contain any of these characters: `/`, ` `, `:`, `#`. ### `dependencies` @@ -52,7 +52,7 @@ The simplest form for this section is a list of `str` encoding `MatchSpec`-compa This section can also contain "subsection" dictionaries that map `str` to arbitrary values. Each key refers to a non-conda installer tool that will process the associated contents as necessary. The additional subsections MUST be processed after the conda requirements. They SHOULD only add new contents. These keys SHOULD NOT overwrite existing contents. -Currently known subsections include `pip`, the contents of which encode a list of `str` referring to PyPI requirements. How to process this list is left as an implementation detail, but common approaches involve invoking the `pip` command-line directly. +Currently known subsections include `pip`, the contents of which encode a list of `str` referring to PyPI requirements. How this list is processed is left as an implementation detail, but common approaches involve invoking the `pip` command-line directly. Additional subsections are allowed. The conda client MUST error out if it cannot process unknown sections. @@ -62,7 +62,7 @@ Optional, `list[str]`. These are the conda channels that will be queried to solve the requirements added in `dependencies`. They can be expressed with names, URLs and local paths. -If not specified, the conda client MUST use the default configuration. When specified, these channels MUST be used to populate the channel list passed to the solver. Then default channel configuration MUST be appended, unless the special name `nodefaults` is present. When this is the case, the default configuration MUST no be appended. The `nodefaults` name MUST not be passed to the solver. +If not specified, the conda client MUST use the default configuration. When specified, these channels MUST be used to populate the channel list passed to the solver. Then default channel configuration MUST be appended, unless the special name `nodefaults` is present. When this is the case, the default configuration MUST no be appended. The `nodefaults` name MUST NOT be passed to the solver. ### `variables`
Title Specification of `environment.yml` input files
Title Specification of environment.yml input files
Status Draft
Author(s) Jaime Rodríguez-Guerra <jaime.rogue@gmail.com>
Created June 4, 2024