Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify spec requirements: section style, structure, & compiler behavior #255

Open
cisaacstern opened this issue Sep 22, 2024 · 0 comments

Comments

@cisaacstern
Copy link
Collaborator

#225 implements a first pass on #241. Here is an example of what the current draft implementation looks like in practice:

requirements:
  compile:
    - name: ecoscope-workflows-core
      version: "*"
      channel: file:///tmp/ecoscope-workflows/release/artifacts/
  run:
    - name: ecoscope-workflows-core
      version: "*"
      channel: file:///tmp/ecoscope-workflows/release/artifacts/
    - name: ecoscope-workflows-ext-ecoscope
      version: "*"
      channel: file:///tmp/ecoscope-workflows/release/artifacts/

There are (at least) three follow-ons we should address before considering this API stabilized (in increasing order of complexity):

  1. Style. Mappings of mappings (as opposed to a list of mappings) would be closer to how pixi.toml conveys the same information and perhaps more readable, e.g.
 requirements:
    ...
    run:
      ecoscope-workflows-core: { version: "*" , channel: file:///tmp/ecoscope-workflows/release/artifacts/ }
      ecoscope-workflows-ext-ecoscope: { version: "*", channel: file:///tmp/ecoscope-workflows/release/artifacts/ }
  1. Structure (i.e. what fields are included). I currently have a distinction between compile and run dependencies as a nod to the general build/run distinction seen in pyproject.toml and rattler-build recipe.yaml, etc. but I should note as it stands the compile: section is a placeholder and is not used for anything. ⚠️ ... In reality, I found, both packages are needed at compile-time (because we need the task names and signatures registered from ext-ecoscope to compile) ... but we don't need the runtime dependencies of ext-ecoscope (e.g. geopandas, etc.) to compile (since those imports are deferred). Another way of saying this is we need all extensions at compile time, but they can be installed in --no-deps mode (or equivalent) ... because by definition their only import dependency must be ecoscope-workflows-core. This suggests we may not need a compile/run table distinction, and instead we can just have a single top-level collection of requirements: but which are handled differently depending in compile vs. run contexts. (In the former, extensions are installed with --no-deps, or an equivalent "no-deps" variant package via conda; and the latter, with all dependencies.)
  2. Behavior. This is the hardest thing and the question I have is how do we want to handle installing the extension packages at compile time, to ensure we have the right environment to do so. (The fundamental question addressed by pyproject.toml's build-system.requires, rattler-build requirements.build, etc.). A pretty tractable possibility is that our own CLI just inspects the requirements table and uses something like https://conda.github.io/rattler/py-rattler/installer/ to build a temporary environment from the dependencies listed there. To compile in. This means that, as long as the extensions listed are 1️⃣ trusted and 2️⃣ available via a conda channel, then we can do this dynamically (just like hatchling, rattler-build, setuptools, etc.) does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant