Switch to flit

[flying-sheep]

What stays the same:

• pip install scanpy
• pip install .
• pip install git+https://...
• you can install your deps with conda
• you can do a dev install

What changes:

• Please check the install docs, in short:
  • pip install -e .[every,single,extra] → flit install -s for dev installs
  • beni pyproject.toml > environment.yml for conda
• Extremely simple flit build and flit publish. Maybe install keyring to store your publish password, and you know everything you need to.
• flit build doesn’t clutter your dev directory with build/ and *.egg-info/ junk, it just creates dist/scanpy-*{.whl,.tar.gz}.
• No more obscure stuff nobody understands (MANIFEST.in, package_data, …)
• Centralized setup configuration in pyproject.toml instead of spread over multiple files

[flying-sheep]

Current problems:

• flit install --pth-file --deps=production doesn’t work with setuptools-scm (just in conda?)
  • it needs setuptools_scm, which is in the dev extra. We need to document this.
  • circumvent working_set stale when installing distributions after importing pkg_resources pypa/setuptools#2531
• flit doesn’t work if setup.py exists (still? where’s the issue?)
• pip install -e with the setup.py doesn’t install deps if the metadata is broken.

Not a problem (to my knowledge), but because you mentioned it:

conda doesn’t identify flit installed distributions as <develop>. This is because flit installs a regular .dist-info directory instead of dumping an .egg-info into your dev directory and adding an .egg-link file to the site-packages
That needs to be fixed by conda, I suggest they should parse symlinks.

[ivirshup]

flit install --pth-file --deps=production doesn’t work with setuptools-scm (just in conda?)

Fix: it needs setuptools_scm, which is in the dev extra. We need to document this.

I had run into a problem with

 flit install --pth-file --deps=develop

[flying-sheep]

I figured it out: pypa/setuptools#2531

[flying-sheep]

Due to the first problem (that blocked using flit in all circumstances) being fixed and the last two standing in the way of the setup.py working, I think flit-only seems more likely at the moment.

@ivirshup what do you think?

PS: here’s some progress on standardized --editable mode pypa/pip#8212

[ivirshup]

I'm not comfortable with doing an absolute switch without significant testing.

I would want a few frequent contributors (Fidel, Sergei, Goecken?) to try it out, try and break it, and make sure we could get things done.

I also still have concerns about the limitations of what flit can distribute, and would like to hear other's thoughts on this.

If a setuptools based workflow can happen, then I don't think these steps are necessarily blocking.

[flying-sheep]

Ha, got it! Seems like all tools break differently on multiline strings in the metadata.

I changed the string back to single ticks to test that metadata problem and then forgot. Now it works and is tested!

Also I checked and flit build seems to be working with setup.py existing?

[Zethson]

Huh, very interesting. Why did you choose flit over https://python-poetry.org/ ?
The Theislab may soon be using https://github.com/cookiejar/cookietemple/tree/development whose cli-python template is using Poetry. Also, Poetry is much more popular (>10x more popular).

[flying-sheep]

Poetry is great! But i remember two problems:

1. no good way to editably install into some env: pip install -e . equivalent? python-poetry/poetry#34
2. doesn’t support plugins yet so only hardcoded versions in static metadata: Support automatic versioning setuptools_scm-style? python-poetry/poetry#140

Maybe @ivirshup knows more. We talked about it way back when.

The things I’m missing from flit are better dynamic version support (currently a bit hacky, but a PR exists) and sth. like poetry install --no-root.

[ivirshup]

I've found poetry mostly a pain the last few times I've tried it. Maybe it's gotten better, but the poetry.lock file and use of virutalenvs seemed more restrictive than useful. My impression was that it was designed to solve problems for a very different use case than scientific computing.

I like that flit is much more limited in scope, and does not try to do everything.

I'd also be worried using poetry would hamper contributions from people unfamiliar with it, and I don't think bioinformaticians are going to be familiar with it.

[Zethson]

All right, fair points.

Poetry is great! But i remember two problems:

no good way to editably install into some env: python-poetry/poetry#34
doesn’t support plugins yet so only hardcoded versions in static metadata: python-poetry/poetry#140

I also stumbled upon the editably install issue. This is not an issue that Poetry can solve at the moment as explained in the thread. I do however understand that this is an issue for scanpy (considering the strong anndata dependency etc).

Regarding plugins - they are on the roadmap and should appear at some point. Considering that the community is very active whereas the main developer is not anymore this may solve the editable install with some hack as well.

I agree with your points and Poetry is not yet the solution that we should currently use, but I think it is the proper solution that we should aim for.

I'd also be worried using poetry would hamper contributions from people unfamiliar with it, and I don't think bioinformaticians are going to be familiar with it.

I don't think that anybody is familiar with flit either, but it is slightly less intrusive and does not fundamentally change so many things like Poetry does. However, many things that Poetry does change make a lot of sense and solve other issues that we did not discuss here yet.

So yeah, I personally would wait for Poetry to get it's plugin system and for the editable install issue to get a proper PEP. But if you don't feel like waiting Flit might be fun :)

Cheers

[flying-sheep]

Flit has a very tiny surface area. You can learn its full CLI in literally 2 minutes, as it doesn’t include any kind of new concept (like Poetry’s venv management).

[Zethson]

Yeah, I agree. Go for it ;) Saturday, 23 January 2021, 11:48AM +01:00 from Philipp A. [email protected] :

…

Flit has a very tiny surface area. You can learn its full CLI in literally 2 minutes, as it doesn’t include any kind of new concept (like Poetry’s venv management). — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or unsubscribe .

[flying-sheep]

@Zethson as asked in your PR, can you please add a commit to fix the conflict here?

[ivirshup]

@flying-sheep, just gave this another try and ran into problems with the symlink installation. Here's what's happening

# Create environment
conda create -yn flit-symlink-test python=3.8 flit gh
conda activate flit-symlink-test

# Get scanpy and install
git clone https://github.com/theislab/scanpy.git
cd scanpy
gh pr checkout 1527
flit install -s --deps=develop

# Check that install worked
cd ..
python -c "import scanpy; print(scanpy.__version__)"
# 1.7.0rc2.dev12+ga760655a

# Check that code is modified when I change version
cd scanpy
git checkout 1.7.0
cd ..
python -c "import scanpy; print(scanpy.__version__)"
# 1.7.0rc2.dev12+ga760655a

The second attempt to print the scanpy version should print 1.7.0.

This problem appears to be specific to the symlink install. If I install the package with flit install --pth-file --deps=develop or pip install -e ".[dev, test] the correct version is printed.

It does look like scanpy is being symlinked correctly:

$ python -c "import scanpy; print(scanpy.__file__)"   
/Users/isaac/miniconda3/envs/flit-symlink-test/lib/python3.8/site-packages/scanpy/__init__.py
$ python -c "import scanpy; from pathlib import Path; print(Path(scanpy.__file__).resolve())"
/Users/isaac/Documents/Issues/scanpy_flit/manual_test/scanpy/scanpy/__init__.py

And, interestingly, the version is reported correctly if we are in the scanpy directory

cd scanpy
python -c "import scanpy; print(scanpy.__version__)"
# 1.7.0

---

As a side note, I've been using a little bit of scripting to help automate testing this PR. It's in julia, but I'll include that here incase it's helpful to the PRs progress.

using Base.Iterators

install_dict = Dict([
    ("flit-pth", `flit install --pth-file --deps=develop`),
    ("flit-sym", `flit install -s --deps=develop`),
    ("pip-edit", `pip install -e ".[dev, test]"`),
])
names = collect(keys(install_dict))
setup_url = "https://gist.githubusercontent.com/flying-sheep/9e7776f4cbe967397702e1c581e3a40a/raw/a2707597abf90f14efdb37751bdd2330cc4d1359/setup.py"

install_results = Dict{String, Tuple{String, String}}()
import_results = Dict{String, Tuple{String, String}}()

# settings = collect(product(install_dict, (true, false)))
# names = map(settings) do ((install_type, install_command), w_setup)
#     w_setup ? "$(install_type)_setup" : install_type
# end


function create_envs!(names)
    procs = map(
        name -> run(`conda create -yn $(name) python=3.8 flit`, wait=false),
        names
    )
    while !all(process_exited.(procs))
        sleep(0.1)
    end
    @assert all(success.(procs))
    procs
end

function delete_envs!(names)
    procs = map(
        name -> run(`conda env remove -yn $(name)`, wait=false),
        names
    )
    while !all(process_exited.(procs))
        sleep(0.1)
    end
    @assert all(success.(procs))
end

function setup_directory!(name, w_setup=false)
    if isdir(name)
        rm(name, recursive=true)
    end
    mkdir(name)
    cd(name) do
        run(`git clone https://github.com/theislab/scanpy.git`, wait=true)
        cd("scanpy") do 
            run(`gh pr checkout 1527`)
            if w_setup && !isfile("setup.py")
                run(`wget $(setup_url)`)
            end
        end
    end
end

function read_permissive(cmd)
    stdout = IOBuffer()
    stderr = IOBuffer()
    proc = run(pipeline(cmd; stdout, stderr), wait=false)
    wait(proc)  # hmm
    String(take!(stdout)), String(take!(stderr))
end

function install!(name, install_command)
    cd("$(name)/scanpy") do
        read_permissive(`conda run -n $(name) $(install_command)`)
    end
end

env_contents(name) = read(`conda env list -n $(name)`, String)

function check_version(name)
    read_permissive(`conda run -n $(name) python -c "import scanpy; print(scanpy.__version__)"`)
end

function checkout_ref!(name, ref)
    cd("$(name)/scanpy") do
        run(`git checkout $(ref)`)
    end
end


# setup
# delete_envs!(names)  # If already made
create_envs!(names)

# Create directories
setup_directory!.(names)

# Try to install

for (name, cmd) in install_dict
    println(name)
    out, err = install!(name, cmd)
    if !isempty(err)
        println(err)
    end
    print("\n\n")
end

check_version.(names)

checkout_ref!.(names, "1.7.0")

check_version.(names)

[ivirshup]

A few more comments (and code review)

---

For the .azure-pipelines conflicts, could you apply your changes on top of the master version of that config?

Also, could the "Build and twine check" job build with flit?

[flying-sheep]

Time to merge?

[ivirshup]

I don't actually use poetry to create the development environment.

@adamgayoso, I'd be interested to hear about what you like about poetry in this case. My impression, from skimming python packaging flame wars on HN, was handling the development environment was poetry's whole thing.

[ivirshup]

Okay all done. flit install -s was getting too messy as some dependency installs scanpy and then things can’t be symlinked …

Can you elaborate on this? This sounds like it could be a more general problem.

When I try this:

$ conda create -yn "test-flit-sym" python=3.8 flit
$ conda activate test-flit-sym
$ flit install -s --deps=develop
$ pip install scvelo
...
  Attempting uninstall: scanpy
    Found existing installation: scanpy 1.7.0rc2.dev33-gf953fbc8.d20210218
    Uninstalling scanpy-1.7.0rc2.dev33-gf953fbc8.d20210218:
      Successfully uninstalled scanpy-1.7.0rc2.dev33-gf953fbc8.d20210218
Successfully installed loompy-3.0.6 numpy-groupies-0.9.13 scanpy-1.7.0 ...

This seems like pretty bad behavior for a development environment. We definitely don't want the dev install to be uninstalled when a new package gets downloaded.

Why is this even happening, and is it fixable?

Less importantly, if you then run flit install -s --deps=develop, the version reported by pip does not get updated.

$ flit install -s --deps=develop
$ pip list | grep scanpy
scanpy                        1.7.0

It looks like it actually does get overwritten, but it's non-obvious.

---

better leave the pip install -r in temporarily

Why not just use pip install -e here? I'm a bit concerned that pinning pip to an old version could lead to problems, especially while pip is going through a lot of changes.

---

Quick recap of things remaining (summary for myself)

• Exclude setup.py from sdist using the standard way, not via .gitignore
• flit mangles the build version part of wheel filenames, in a way that pip just started checking for. We can work around this in CI, and should not affect distributions (as there is no build info there). I believe this will break pip install git+https:{repo} package from the repo. pip install -e still works.
• flit symlinked packages seem to be overwritten if a new package is installed which has the symlinked package as a dependency.
• There is a fairly large workaround to make the package version available if the dependencies are not installed. Is it possible to use something more standard like versioneer here?

[flying-sheep]

This seems like pretty bad behavior for a development environment. We definitely don't want the dev install to be uninstalled when a new package gets downloaded.

Well, scvelo depends on 1.7 and you have a release candidate of that one installed, so what happened is the only correct behavior: It uninstalled an incompatible version to install a compatible one.

If your install’s metadata was outdated and it was in fact a compatible one, then you forgot to refresh the metadata by reinstalling it. That’s annoying but necessary as editable installs are nonstandard and therefore not well integrated into how package metadata works.

Why not just use pip install -e here?

Because development installs in general are nonstandard, and pip install -e in particular uses the deprecated setup.py.

Tasks

• Exclude setup.py from sdist using the standard way, not via .gitignore

sounds good!

I'm a bit concerned that pinning pip to an old version could lead to problems, especially while pip is going through a lot of changes.
...

• flit mangles the build version part of wheel filenames, in a way that pip just started checking for.

No, as far as I can see, pip arbitrarily decided to not allow local version specifiers in wheel filenames. AFAIK nothing says there can’t be pluses in there, only that you can’t upload packages with local specifiers in their version to PyPI. Which we don’t do here, so pip should chill.

If flit decides to work around that quirk, or pip relaxes, we can unpin pip.

• flit symlinked packages seem to be overwritten if a new package is installed which has the symlinked package as a dependency.

Seee above. Has nothing to do with flit. What made you thing that anyway?

• There is a fairly large workaround to make the package version available if the dependencies are not installed. Is it possible to use something more standard like versioneer here?

No. Either we hardcode a string constant in the __init__.py or we leave it like it is until flit allows an alternative.

That’s the only disadvantage flit has IMHO, but we discussed that at length in the past and found it to not be a problem as the hack is robust and well documented.

[ivirshup]

Well, scvelo depends on 1.7 and you have a release candidate

scvelo depends on scanpy>=1.5, so I don't think this is the cause. Do you know why this is happening, and can you provide counter examples where it doesn't happen?

If your install’s metadata was outdated and it was in fact a compatible one, then you forgot to refresh the metadata by reinstalling it.

I'm not sure what you mean by this. Does flit install -s --deps=develop not count as reinstalling? Are you counting flit install -s as a development install?

Because development installs in general are nonstandard, and pip install -e in particular uses the deprecated setup.py.

I think pinning pip to an old version is worse than using a common, even if non-standard, installation method.

No, as far as I can see, pip arbitrarily decided to not allow local version specifiers in wheel filenames.

My reading of the PR in flit and the subsequent discussion in the pip and PEP threads suggests to me that the issue is pip validating the metadata name against the wheel, while a spec exists saying the wheel can't contain characters that are allowed by version specs. If anything, pip suddenly started expecting exact version specifiers in wheel filenames, while a spec exists that says how the filenames should be mangled. flit did the mangling, and pip now says that's wrong. It looks like the direction the discussion is headed is PEP 427 is wrong, and pip is right. I have no idea what sort of timeframe should be expected here.

[flying-sheep]

scvelo depends on scanpy>=1.5, so I don't think this is the cause. Do you know why this is happening, and can you provide counter examples where it doesn't happen?

Ah weird. Pip tries to resolve the dependencies, and for that purpose gets all the installed packages’ metadata, then tries to figure out a configuration of upgrades that makes things work. No idea why it sees “1.7.0rc2” and decides “I’ll update this even when not asked to update”. Maybe raise this issue with pip?

I'm not sure what you mean by this. Does flit install -s --deps=develop not count as reinstalling? Are you counting flit install -s as a development install?

Yes, that’s a reinstall in some development mode. My point was that if a scanpy pre-1.7 version really was installed, maybe pip was correct to update to 1.7 for some reason. However since we’re past 1.7 now, unless you haven’t git-pulled yet, I assume your dev install’s metadata has gone stale. I guess pip wouldn’t uninstall your dev install if you had run git pull && flit install -s, therefore updating the metadata. But I could be wrong, as I have no idea why pip thought it necessary to touch scanpy when installing scvelo.

I think pinning pip to an old version is worse than using a common, even if non-standard, installation method.

Our setup.py is a compatibility shim solely for fallback use, not something to be relied upon in any part of our process.

Usually when something does an arbitrary change making our life harder, our approach is pinning it temporarily until it fixed that or the infrastructure has adapted to its whims, right?

It looks like the direction the discussion is headed is PEP 427 is wrong, and pip is right.

Accepted PEPs are specs, so only pip and flit can be right or wrong (as they implement it). If people decide that what pip does happens to be better than the currently spec-compliant behavior, the spec can be changed accordingly. Until then pip is wrong, so we should pin its version to one that accepts spec-compliant behavior. (we can change our approach if that happens to drag on too long)

I see you already commented in pypa/pip#9628, so I guess that’s the better place for following that.

[ivirshup]

symlink installs being uninstalled (most important)

No idea why it sees “1.7.0rc2” and decides “I’ll update this even when not asked to update”. Maybe raise this issue with pip?

Yeah, this is super weird. I think it's also blocking for adopting flit as recommended way to install scanpy to a dev environment. I also raised this on the call yesterday, and I don't think anyone disagreed with this assessment.

I see two paths forward here:

• You're able to solve this in this PR
• We merge mostly as is, but we add back pip install -e as a way to make a development environment, and add a .. note to the flit installation instructions warning people about this behaviour. I would also want a commitment from you to look into this issue.

Pinning Pip on CI

Usually when something does an arbitrary change making our life harder, our approach is pinning it temporarily until it fixed that or the infrastructure has adapted to its whims, right?

I still have the concern that pinning pip to an old version could lead to problems, especially while pip is going through a lot of changes. But we can leave this for now. If getting this wheel issue solved drags on for multiple pip versions, we may need to reconsider.

PEP stuff

I see you already commented in pypa/pip#9628

I think that conversation is happening in multiple places, so might be hard to track.

Installing from the repo

As it stands:

conda create -n scanpyenv python=3.8
https://github.com/theislab/scanpy.git
cd scanpy
pip install .

Will error, unless the commit at the tip of master happens to be tagged with a release version. Right now I don't think this is an issue since I wouldn't expect anyone to install from github unless they were setting up a development environment. And if they are setting up a dev environment, they should be using pip install -e or flit install -s.

I'm not 100% confident this isn't an issue, and it would be good to get more opinions on this.

Version resolution

No. Either we hardcode a string constant in the init.py or we leave it like it is until flit allows an alternative.

That’s the only disadvantage flit has IMHO, but we discussed that at length in the past and found it to not be a problem as the hack is robust and well documented.

On how version strings are handled/ generated:

I would be more comfortable using a solution that other packages used too. In particular, this looks very brittle to me:

    for frame in traceback.extract_stack():
          if frame.name == 'get_docstring_and_version_via_import':
              return True

I don't see why flit couldn't just change the name of a function that is called internally at any point. I also think that at the moment, you and I are the only contributors who would have any idea what was going on if this acts up.

[ivirshup]

I believe the issues are related. I think pip is trying to get the version of the wheel by parsing it's name instead. For example, I just ran into this with where scanpy's version was being reported as: 1.8.0.dev34-g8d8039d9 – which I don't think is a valid version. I then pip install -e'd and got a version of 1.8.0.dev29+gc8309488 (different branches) and it worked fine.

[flying-sheep]

Damn more merge conflicts. I’ll fix it one last time then merge this, OK?

Unfortunately this needs to be a rebase, otherwise I have to resolve everything again when rebasing.

[flying-sheep]

Yeah, this is super weird. I think it's also blocking for adopting flit as recommended way to install scanpy to a dev environment

How has that to do with flit? Will pip just block upgrades things it identifies as “editable installs” from being upgraded while happily upgrading “normally installed” release candidates?

I still have the concern that pinning pip to an old version could lead to problems, especially while pip is going through a lot of changes. But we can leave this for now. If getting this wheel issue solved drags on for multiple pip versions, we may need to reconsider.

Exactly!

I'm not 100% confident this isn't an issue, and it would be good to get more opinions on this.

It wasn’t an issue for the year this PR has lingered, and now it’s pypa/pip#9628 and related discussions (which are already converging)

I would like to meaningfully contribute to scanpy again instead of having to fix merge conflicts in this and discussing what pip broke this time.

In particular, this looks very brittle to me:

It isn’t, as you agreed on like 8 months ago. Worst that happens is “ImportError: cannot import ‘foo’” when building, which we can fix in 5 minutes.

[ivirshup]

I would also like to see this merge. I've put a lot of time and effort into reviewing it, so we can get this over and done with.
Your contributions are invaluable to the project, and I'd really like to see you contributing to other things.

The reason I'm so hard on this is that it's critical to our project (and getting new contributors), and it's a part of the stack I don't understand.
I think you're the only one on the team who has a lot of understanding of the packaging ecosystem. The practical effect of this is that when things around this break, most of us have no idea what could be going wrong. What we have on master right now pretty much works. We've run into issues before, but it's been a while. Right now it's pretty smooth to set up a dev environment and contribute.

It isn’t, as you agreed on like 8 months ago.

I don't recall this specifically from 8 months ago. Theres a good chance that because I don't have as great of a knowledge about how packaging works, I understood our conversation in a different way.

Because this is new, there's definitley going to be bugs. These are bugs with part of the stack that we don't have a lot of expertise in, so I'd like to minimize these before they become blockers.

$ conda create -yn flit-deps python=3.8 flit
$ conda activate flit-deps
$ flit install -s --dep=develop  # Make development install of scanpy
$ pip install scvelo  # Install project that depends on scanty
...
  Attempting uninstall: scanpy
    Found existing installation: scanpy 1.8.0.dev49-ge715cd98
    Uninstalling scanpy-1.8.0.dev49-ge715cd98:
      Successfully uninstalled scanpy-1.8.0.dev49-ge715cd98
...
# Development version of scanpy has now been uninstalled

This is bad, and should not be the default experience for people who want to contribute. This does not give an error, or a warning. Yes there are solutions being proposed upstream, but we don't know how long until they are implemented.

Here's what I propose. I think this can be merged basically as is. However, until these issues are resolved: development installation instructions has to have pip install -e listed, and there has to be a note saying flit -s installations will be overridden due to a bug in pip. This stuff can be removed once this is fixed upstream.

[flying-sheep]

I would also like to see this merge. I've put a lot of time and effort into reviewing it, so we can get this over and done with.
Your contributions are invaluable to the project, and I'd really like to see you contributing to other things.

thank you, I really appreciate this ❤️

The reason I'm so hard on this is that it's critical to our project (and getting new contributors), and it's a part of the stack I don't understand.
I think you're the only one on the team who has a lot of understanding of the packaging ecosystem. The practical effect of this is that when things around this break, most of us have no idea what could be going wrong. What we have on master right now pretty much works. We've run into issues before, but it's been a while. Right now it's pretty smooth to set up a dev environment and contribute.

Totally understood. My motivation to use flit is that it’s simpler and therefore better both for first-time contributors (to get started) and experienced people (to debug), whereas CLI, metadata, and code of setuptools/pip is very complex and a nightmare to debug.

I know that due to flit being used less, there needs to be someone who understands the packaging ecosystem to fix things when they’re broken instead of cargo-culting one of the million answers around setuptools on StackOverflow.

Here's what I propose. I think this can be merged basically as is. However, until these issues are resolved: development installation instructions has to have pip install -e listed, and there has to be a note saying flit -s installations will be overridden due to a bug in pip. This stuff can be removed once this is fixed upstream.

OK, will do! Can you link me tp the upstream discussion of this problem please?

[ivirshup]

I've opened an issue to track the flit install -s problem. It's at: pypa/flit#394, but I figure it might get merged with the wheel name issue since they seem related.

[ivirshup]

I think this plus the change to sphinx pinning and this should be good to go.

Also, release note if you want?

---

Is the anndata PR up to date as well?

---

Then, feel free to merge this.

🎆 🎆 🎆

[flying-sheep]

Let’s wait for #1700

/edit: done!

[flying-sheep]

OK, merging! Let’s do anndata ASAP then so people don’t have two different dev environments for too long.

[flying-sheep]

Had to restore the branch to reopen #1700 (and switch its base to master)