Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated dependency to pin everything w/ pip-compile #82

Merged
merged 5 commits into from
Sep 6, 2022
Merged

Conversation

domna
Copy link
Member

@domna domna commented Sep 2, 2022

According to #72 I wanted to introduce a better dependency management. As pipenv is very inconsistently released and poetry seems to be a whole project management suite with publishing, env creation etc. which I found a little overboard I opted for the nice pip-compile method which just creates a pinned requirements.txt from a requirements.in file (the requirements.in file is basically the previous requirements.txt file).

Here's the link to pip-tools.

@MarJMue Do you like this solution, too?

@domna domna requested a review from MarJMue September 2, 2022 18:25
@domna
Copy link
Member Author

domna commented Sep 2, 2022

Okay, it actually seems not to be too easy for different python versions :) I just updated to use requirements.in for the actions.

Here's the thread from pip-compile on the topic: jazzband/pip-tools#639
I'm not yet exactly sure how to resolve this to build reproducible installs.

@MarJMue
Copy link
Collaborator

MarJMue commented Sep 2, 2022

There is also a paragraph here

@MarJMue
Copy link
Collaborator

MarJMue commented Sep 2, 2022

I like the general method to get reproducible builds, but i am not so sure how practical it is, if other packages in your enviroment have other versions of a package pinned.

But it is still way better to provide a tested build, then to assume that everything is fine with newest versions (see #80)

@MarJMue
Copy link
Collaborator

MarJMue commented Sep 3, 2022

After thinking a bit more about it, i am all for it and we should use pip-compile to generate known working enviroments for each python version. We can still hint to the requirements.in without any guarantee that it will work.

The main branch can continue to be tested with current versions and we will pin dependencies at fixed points, most likely every minor release.

@domna
Copy link
Member Author

domna commented Sep 3, 2022

Yes, I just found out that we can ditch the requirements.in completely, as pip-compile can also just get the requirements from setup.cfg. I'm still figuring out how to deal with the extra requirements (which is just plot for us), because I think it would be nice to have three requirements, one for dev (with testing etc.) maybe named dev-requirements.txt (containing base + plot + testing) one for the extra plot requirements (extra-plot-requirements.txt or plot-requirements.txt) and one base requirements.txt. It would be nice to generate these three directly from setup.cfg I think.

Edit: Ok just found the --extra ... option for pip-compile which takes the extra requirements in.

@domna
Copy link
Member Author

domna commented Sep 3, 2022

The last remaining question for me is how to easily generate these requirements files. We could either just write a bash script for doing it or use tox for it (which seems to be a really nice tool, but I'm still figuring out how to use it properly). Tox gives us also easy access to different python versions and we could also do tests for any version locally (... and then use tox in the action to reproduce the exact same environment 😎)

Here's a nice tutorial on tox.

@domna
Copy link
Member Author

domna commented Sep 3, 2022

Some more thoughts on requirements for different python versions.
Since users of the package will mostly install via pip they will only use the dependencies from setup.cfg so they do not depend on the latest requirements file. Our action would ensure that this still installs.

So mostly the requirements file will be used for people developing or for reproducing an exact environment (i.e. in docker). Therefore, I would not feel too badly about only supporting one or maybe two main python versions we support there.

@MarJMue
Copy link
Collaborator

MarJMue commented Sep 3, 2022

The last remaining question for me is how to easily generate these requirements files. We could either just write a bash script for doing it or use tox for it (which seems to be a really nice tool, but I'm still figuring out how to use it properly). Tox gives us also easy access to different python versions and we could also do tests for any version locally (... and then use tox in the action to reproduce the exact same environment 😎)

Here's a nice tutorial on tox.

Tox seems very nice for the job. Or we just run a manual github action with the respective python version and do a pip-compile.

@domna domna merged commit d4faee1 into master Sep 6, 2022
@domna domna deleted the pip-tools branch September 6, 2022 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants