Add constructor manager cli #34

goanpeca · 2022-10-05T15:14:07Z

Related to #10

Overview

Constructor based installers

These installers will contain 3 main environments when created:

A base environment which will contain:
- conda (package and environment manager)
- mamba (conda "replacement" for performing faster solves, transactions and downloads)
- conda-lock (create locked and reproducible environments)
A constructor-manager environment which will contain:
- constructor-manager-backend (this PR)
- constructor-manager-client (Add constructor manager api #46)
- constructor-manager-ui (Update UI and connect to constructor manager API #63)
An application environment, using the conventions <package-name>-<version>, for the example image above, napari-0.4.15

More information on packaging on https://napari.org/dev/developers/packaging.html#constructor-based-installers

constructor-manager-backend

This pull request is in charge of creating the constructor-manager-backend package which in its present form is a CLI utility that will work under the hood with conda, mamba and conda-lock (living in the base environment) to provide the following functionalities in the creation of bundle applications:

The backend can be called using the CLI program constructor-manager and the following actions are available

check-updates: Query for new updates available for the managed application by providing one or more anaconda.org channels. By default we query the conda-forge channel located at https://anaconda.org/conda-forge/
update: update a current application to a new version of it. This process will create a new conda environment following the convention <package-name>-<new-version>, create new menu shortcuts (using the menuinst branch https://github.com/conda/menuinst/tree/cep-devel/menuinst), create a new restore point (using conda-lock), remove the old environment and the corresponding shortcuts for the old versions of the managed application.
revert: this will revert the current application to the previously installed version on the computer if a restore point is found. This will follow a similar process of creating a new conda environment with the convention <package-name>-<old-version>.
restore: similar to revert but restore to a previously found state of the current version. This command could become a single one by providing the specific restore file (which is created by conda lock)
lock-environment: create a lock file of the current state of the application environment. This can be called by the client applications (using constructor-manager-client) to check for changes in the environment. If no changes are detected, no lock is created.
uninstall: to be implemented
get-status: get information on a currently running update/restore/revert in progress.

Some of these commands can be run in parallel others create a lock to prevent multiple instances of an update/restore/revert process.

More information on the README of the package

Work progress

Future work

Handle multiple applications
Provide rich information for debugging
Provide background update within the running application

goanpeca · 2023-03-22T15:02:27Z

@dalthviz, @aganders3 this one is ready for review, except for the file insttaller.oy which I am simplifying a lot since we do not need to use a queue and the other abtsractions.

(also fixing the windows test)

Thanks 🚀

dalthviz

Hi @goanpeca ! Left some initial comments/questions mostly on the README file, the menuinst dependency and some commented out code that I found lying around. Tried checking some of the commands but I got some tracebacks with the check-updates one after trying to setup a napari installation to be detected (before that the command was working and it detected I didn't have napari installed 👍). Maybe I made some mistake trying to setup things 😅 but I left a comment just in case with some initial debugging I did.

dalthviz · 2023-03-22T15:24:12Z

.pre-commit-config.yaml

+  # - repo: https://github.com/pycqa/isort
+  #   rev: 5.11.5
+  #   hooks:
+  #     - id: isort
+  #       name: isort (python)


Maybe this could be removed?

Yes isort can be replaced by ruff --fix with the I rules configured, but I don't think they're part of the defaults.

dalthviz · 2023-03-22T15:50:20Z

constructor-manager/README.md

+- mamba
+- packaging
+- requests
+- pyyaml


Should menuinst be listed as a requirement here too?

Yes, but we are relying on a fork so in the meantime it should install that fork. Will add a line

dalthviz · 2023-03-22T15:53:01Z

constructor-manager/README.md

+### Menuinst
+
+```bash
+pip install git+https://github.com/conda/menuinst.git@cep-devel
+```


Maybe this could be better at the top of the README with the rest of the instructions to install?

dalthviz · 2023-03-22T15:56:34Z

constructor-manager/README.md

+Check for the version of the currently (latest?) installed package.
+
+```bash
+constructor-manager check-version "napari"


I think this and the commands bellow need to use constructor-manager-cli instead of constructor-manager since that's how the entrypoint is still declared at setup.cfg or maybe setup.cfg needs to be updated?

dalthviz · 2023-03-22T16:38:04Z

constructor-manager/src/constructor_manager/__init__.py


-__version__ = "0.0.1"
+__version__ = "0.1.0"
+VERSION_INFO = (0, 1, 0)


Is this VERSION_INFO constant needed?

Not really I think 🤔

dalthviz · 2023-03-22T17:30:37Z

constructor-manager/src/constructor_manager/main.py

+    result = []
+    if args.command in _COMMANDS:
+        result = method(**method_kwargs)
+        # time.sleep(5)


Maybe this and the comment below related to time.sleep should be removed?

Yes, that was for testing the lock file

dalthviz · 2023-03-22T17:59:30Z

constructor-manager/src/constructor_manager/actions.py

+
+        packages_original_format = self._installer.list(str(prefix), block=True)
+        for pkg in packages_original_format:  # type: ignore
+            source = "pip" if pkg["platform"] == "pypi" else "conda"


After setting up an env with

conda create -n napari-0.4.15 "napari=0.4.15" "napari-menu=0.4.15" affinder -c conda-forge -y --override-channels

And adding to the conda-meta dir inside the env a .napari_is_bundled_constructor, I tried running:

constructor-manager-cli check-packages "napari=0.4.15" --plugins-url https://api.napari-hub.org/plugins

I got a response error:

{ "data": {}, "error": "Traceback (most recent call last):\n File \"E:\\Acer\\Documentos\\Quansight\\Napari\\packaging-otros\\gonzalo\\packaging\\constructor-manager\\src\\constructor_manager\\main.py\", line 116, in main\n result = _execute(args, lock_file_path)\n File \"E:\\Acer\\Documentos\\Quansight\\Napari\\packaging-otros\\gonzalo\\packaging\\constructor-manager\\src\\constructor_manager\\main.py\", line 65, in _execute\n result = method(**method_kwargs)\n File \"E:\\Acer\\Documentos\\Quansight\\Napari\\packaging-otros\\gonzalo\\packaging\\constructor-manager\\src\\constructor_manager\\actions.py\", line 561, in check_packages\n packages, _ = self._get_installed_packages(prefix, plugins_url)\n File \"E:\\Acer\\Documentos\\Quansight\\Napari\\packaging-otros\\gonzalo\\packaging\\constructor-manager\\src\\constructor_manager\\actions.py\", line 125, in _get_installed_packages\n source = \"pip\" if pkg[\"platform\"] == \"pypi\" else \"conda\"\nTypeError: string indices must be integers\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"E:\\Acer\\Documentos\\Quansight\\Napari\\packaging-otros\\gonzalo\\packaging\\constructor-manager\\src\\constructor_manager\\main.py\", line 121, in main\n sys.stdout.write(json.dumps(data, indent=4))\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\__init__.py\", line 238, in dumps\n **kw).encode(obj)\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\encoder.py\", line 201, in encode\n chunks = list(chunks)\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\encoder.py\", line 431, in _iterencode\n yield from _iterencode_dict(o, _current_indent_level)\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\encoder.py\", line 405, in _iterencode_dict\n yield from chunks\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\encoder.py\", line 438, in _iterencode\n o = _default(o)\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\site-packages\\conda\\__init__.py\", line 145, in _default\n return getattr(obj.__class__, \"to_json\", _default.default)(obj)\n File \"C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\lib\\json\\encoder.py\", line 179, in default\n raise TypeError(f'Object of type {o.__class__.__name__} '\nTypeError: Object of type TypeError is not JSON serializable\n" }

Maybe I'm not correctly setting up the envs but just in case sharing the response. The error seems related with the CondaInstaller list call (self._installer.list) returning {'stdout': ''}. Checked that method and seems like since it runs the list command with mamba it doesn't work if you have installed menuinst from the cep-devel branch. Running mamba list --prefix C:\\Users\\dalth\\anaconda3\\envs\\constructor-manager\\envs\\napari-0.4.15 --json causes:

Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\constructor-manager\Scripts\mamba-script.py", line 6, in <module> from mamba.mamba import main File "C:\Users\dalth\anaconda3\envs\constructor-manager\lib\site-packages\mamba\mamba.py", line 53, in <module> from mamba.mamba_shell_init import shell_init File "C:\Users\dalth\anaconda3\envs\constructor-manager\lib\site-packages\mamba\mamba_shell_init.py", line 8, in <module> from conda.core.initialize import initialize, initialize_dev, make_initialize_plan File "C:\Users\dalth\anaconda3\envs\constructor-manager\lib\site-packages\conda\core\initialize.py", line 71, in <module> from menuinst.knownfolders import get_folder_path, FOLDERID ModuleNotFoundError: No module named 'menuinst.knownfolders'

Yes, I had not tested on woindows so the command is not working. Testing on windows

dalthviz · 2023-03-22T18:06:08Z

constructor-manager/src/constructor_manager/main.py

+    constructor_manager_dir.mkdir(parents=True, exist_ok=True)
+    lock_file_path = constructor_manager_dir / "constructor-manager.lock"
+
+    # result = _execute(args, lock_file_path)


Maybe this should be removed?

dalthviz · 2023-03-22T18:07:00Z

constructor-manager/src/constructor_manager/utils/io.py

+
+@lru_cache
+def get_config_path() -> Path:
+    # path = get_prefix_by_name("base") / "var" / "constructor-manager"


Maybe this should be removed?

dalthviz · 2023-03-22T18:08:15Z

constructor-manager/src/constructor_manager/utils/locking.py

+                        # if not is_spyder_process(int(pid)):
+                        #     raise(OSError(errno.ESRCH, 'No such process'))


Probably this should be removed or updated to detect a napari related process?

Added a note

aganders3 · 2023-03-22T19:29:01Z

constructor-manager/README.md

+constructor-manager check-updates "napari=0.4.16=*pyside*" -c conda-forge
+constructor-manager check-updates "napari=*=*pyside*" -c conda-forge


Maybe add a note (or link?) about the format of this version spec. I was not familiar with the syntax for specifying build strings and wildcards.

Adding extra info

We use a subset of the version spec defined by [conda](https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html) Namely the spec coontains a `<package-name>=<version>=<build-string>` where the `*` symbol can be used as a wildcard.

aganders3

Sorry I'm still going through this, and still haven't run it yet. This looks great so far!

I mostly stopped reviewing when I got to the tests for now, but I'll check those out after I get it installed and do some manual testing.

aganders3 · 2023-03-22T19:48:37Z

constructor-manager/src/constructor_manager/utils/versions.py

+def is_stable_version(version: Union[Tuple[str, ...], str]) -> bool:
+    """Check if a version string corresponds to a stable release.
+
+    Parameters
+    ----------
+    version : tuple or str
+        Version string to check.
+
+    Returns
+    -------
+    bool
+        ``True`` if the version is stable, ``False`` otherwise.
+
+    Examples
+    --------
+    Stable version examples: ``0.4.12``, ``0.4.1``.
+    Non-stable version examples: ``0.4.15beta``, ``0.4.15rc1``, ``0.4.15dev0``.
+    """
+    if not isinstance(version, tuple):
+        version = tuple(version.split("."))
+
+    return not LETTERS_PATTERN.search(version[-1])


I think this will be incorrect for a stable "post release". Is it possible to use the parsed version here and check the is_prerelease property?

Good point, will change!

aganders3 · 2023-03-22T19:50:26Z

constructor-manager/src/constructor_manager/utils/tests/test_versions.py

Maybe add some test cases for post releases in here.

Adding TODOs

aganders3 · 2023-03-22T19:52:54Z

constructor-manager/src/constructor_manager/utils/anaconda.py

+    package_name: str,
+    build: Optional[str] = None,
+    channels: List[str] = [DEFAULT_CHANNEL],
+    reverse: bool = False,


nitpicking: make this boolean a kw-only arg to avoid confusion

I am not sure I understand 🙃

package_name: str, build: Optional[str] = None, channels: List[str] = [DEFAULT_CHANNEL], *, reverse: bool = False,

This?

aganders3 · 2023-03-22T19:54:47Z

constructor-manager/src/constructor_manager/installer.py

+    """Conda installer."""
+
+    def __init__(
+        self, use_mamba: bool = True, pinned=None, channels=(DEFAULT_CHANNEL,)


nitpick: make use_mamba a keyword-only arg

Same as before? 🤔

aganders3 · 2023-03-22T19:58:35Z

constructor-manager/src/constructor_manager/installer.py

+        self._processes = {}  # type: Dict[job_id, subprocess.Popen]
+        self._bin = None  # type: Optional[str]


Update to new-style type annotations.

aganders3 · 2023-03-23T19:51:21Z

constructor-manager/src/constructor_manager/utils/conda.py

+        version = version.rstrip(".*")  # ?
+        version = version.rstrip("*")  # ?


I'm not quite sure what this is meant to do, but rstrip removes all trailing characters in the given list of characters, so this second call is redundant.

Ok, need to fix this˜

aganders3 · 2023-03-23T19:57:03Z

constructor-manager/src/constructor_manager/utils/packages.py

+    str
+        The normalized package name.
+    """
+    return re.sub(r"[-_.]+", "-", name).lower()


huge nitpick as I'm not sure what characters are even valid in package names, but casefold() may be more appropriate than lower() here.

Added a note

aganders3 · 2023-03-23T20:01:12Z

constructor-manager/src/constructor_manager/utils/misc.py

+    new_items: Tuple[Any, ...] = ()
+    for item in items:
+        if item not in new_items:
+            new_items += (item,)
+
+    return new_items


Suggested change

new_items: Tuple[Any, ...] = ()

for item in items:

if item not in new_items:

new_items += (item,)

return new_items

return tuple(dict.fromkeys(items))

I'm not sure it's worth having this separate file for just this function (used once from what I see)?

Agreed, there were more utilities here, but I ended up refactoring them out

aganders3 · 2023-03-23T20:03:48Z

constructor-manager/src/constructor_manager/utils/request.py

+def plugin_versions(
+    url: str,
+) -> List[str]:
+    """Return information on package plugins from endpoint in json.


I guess we will need to document this API generally?

Absolutely, will add the docs

aganders3 · 2023-03-23T20:10:28Z

constructor-manager/src/constructor_manager/utils/shortcuts.py

+        cmd = ""
+        with open(path) as fh:
+            for line in fh:
+                if line.startswith("Exec="):
+                    cmd = line.split("=", 1)[1].strip()
+        if cmd:
+            ret_code = os.system(cmd)


😳

I shouldn't be surprised there's no better way to do this. Apparently gtk-launch should do it but obviously not on every system.

Yes, this is a bit of a "weird part" of the code. Used some suggestions from @jaimergp work on menuinst

goanpeca mentioned this pull request Oct 5, 2022

napari-installation-manager prototype #10

Closed

14 tasks

goanpeca force-pushed the constructor-updater branch from 8932812 to d502ee3 Compare October 11, 2022 17:55

goanpeca changed the title ~~Add initial structure for backend constructor updater~~ Add initial structure for backend constructor manager Oct 26, 2022

goanpeca marked this pull request as ready for review October 26, 2022 21:46

goanpeca force-pushed the constructor-updater branch from 4c1f104 to 5e8e11c Compare October 26, 2022 22:31

goanpeca force-pushed the constructor-updater branch from 911b8c1 to 4b29b90 Compare November 3, 2022 03:50

goanpeca force-pushed the constructor-updater branch from 4b29b90 to c30889d Compare November 28, 2022 00:49

goanpeca mentioned this pull request Nov 29, 2022

Add precommit ci and update gitignore #44

Merged

goanpeca force-pushed the constructor-updater branch from 308f7cc to 09032f0 Compare November 29, 2022 03:18

goanpeca marked this pull request as draft November 29, 2022 03:26

goanpeca mentioned this pull request Nov 29, 2022

Add constructor manager api #46

Open

6 tasks

goanpeca changed the title ~~Add initial structure for backend constructor manager~~ Add constructor manager backend cli Nov 29, 2022

goanpeca self-assigned this Nov 29, 2022

goanpeca mentioned this pull request Nov 29, 2022

Add package stubs and structure #48

Merged

goanpeca force-pushed the constructor-updater branch 2 times, most recently from 778fdf9 to d4e630a Compare November 30, 2022 01:50

goanpeca force-pushed the constructor-updater branch from 28459d2 to 8abe33c Compare December 7, 2022 04:14

goanpeca added 13 commits January 24, 2023 14:54

Add initial structure

d10bfd8

Check for updates

3050e4d

Refactor code and utilities

61d327f

Format code

c454605

Add clean actions

2cad62f

Cli

a342f72

Refactor code

6fc7660

Format code

20dc0d6

Update README

348161b

Update channel management

7033e83

Add locking implementation and format code

57fa3c8

Update typing information

f62013f

Format code and fix types

f4d5930

goanpeca added 3 commits January 24, 2023 14:54

Fix tests

e56bf44

Fix imports

aeb8600

Update comments

b08b4c4

goanpeca force-pushed the constructor-updater branch from 3a9f928 to b08b4c4 Compare January 24, 2023 19:55

goanpeca added 6 commits January 25, 2023 18:04

Update to use an action manager class

0ab9885

Update action manager to work with the API and fix style issues

9c57a76

Add basic logging and update actions

a89ab59

Separate CLI module and clean up locking mechanism

2264544

finish locking mechanism and add logging

efd4588

Update logging and action execution

5c20e61

goanpeca changed the title ~~Add constructor manager backend cli~~ Add constructor manager backend Feb 15, 2023

goanpeca added 4 commits February 22, 2023 11:57

Rename package to backend

54723b3

Add support for delayed updates

a699db2

Rename package to constructor manager

d36c26e

update io imports

014108c

goanpeca changed the title ~~Add constructor manager backend~~ Add constructor manager cli Mar 8, 2023

goanpeca added 2 commits March 7, 2023 22:12

Remove icons and update shortcut handling

8208247

Fix menu creation

be091e9

goanpeca force-pushed the constructor-updater branch from 864717c to f303604 Compare March 15, 2023 21:02

Update CI and fix tests

1e905c4

goanpeca force-pushed the constructor-updater branch from 57c947d to 1e905c4 Compare March 15, 2023 22:35

goanpeca marked this pull request as ready for review March 22, 2023 15:02

goanpeca requested a review from dalthviz March 22, 2023 15:03

dalthviz reviewed Mar 22, 2023

View reviewed changes

aganders3 reviewed Mar 22, 2023

View reviewed changes

aganders3 suggested changes Mar 23, 2023

View reviewed changes

DragaDoncila mentioned this pull request Apr 12, 2023

Update UI and connect to constructor manager API #63

Open

6 tasks

Update code review comments

ff7e9f5

dalthviz mentioned this pull request Jun 26, 2024

Application manager: Decouple from napari #154

Open

3 tasks

		# if not is_spyder_process(int(pid)):
		# raise(OSError(errno.ESRCH, 'No such process'))

		constructor-manager check-updates "napari=0.4.16=pyside" -c conda-forge
		constructor-manager check-updates "napari==pyside*" -c conda-forge

		self._processes = {} # type: Dict[job_id, subprocess.Popen]
		self._bin = None # type: Optional[str]

		version = version.rstrip(".*") # ?
		version = version.rstrip("*") # ?

Add constructor manager cli #34

Are you sure you want to change the base?

Add constructor manager cli #34

Conversation

goanpeca commented Oct 5, 2022 • edited Loading

Overview

Constructor based installers

constructor-manager-backend

Work progress

Future work

goanpeca commented Mar 22, 2023

dalthviz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aganders3 Mar 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dalthviz Mar 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aganders3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goanpeca commented Oct 5, 2022 •

edited

Loading

aganders3 Mar 23, 2023 •

edited

Loading

dalthviz Mar 22, 2023 •

edited

Loading