Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pre_compile fails when installed using setuptools instead of PEP517 #52

Closed
bnavigator opened this issue Jul 14, 2022 · 9 comments
Closed

pre_compile fails when installed using setuptools instead of PEP517 #52

bnavigator opened this issue Jul 14, 2022 · 9 comments

Comments

@bnavigator
Copy link

bnavigator commented Jul 14, 2022 •
edited
Loading

When using the legacy setuptools python setup.py from the PyPI published sdist, setuptools installs a validate_pyproject-*.egg-info (also pre-packaged in the sdist), or validate_pyproject-0.9.dist-info (setup.py bdist_wheel)*. This clashes with pre_compile.load_licenses() which looks for the hard-coded dist_name validate-pyproject. Note the difference in - vs _.

validate-pyproject/src/validate_pyproject/pre_compile/__init__.py

Line 116 in d87a602

"validate_pyproject_license": _find_and_load_licence(_M.files(dist_name)),

validate-pyproject/src/validate_pyproject/__init__.py

Line 11 in d87a602

dist_name = "validate-pyproject"

Workaround: Use a PEP517 installer like pip or build+install.

* Edit: validate_pyproject-0.9.dist-info is also the one installed with a PEP517 frontend and found fine by importlib.metadata.files('validate-pyproject')
@abravalheri
Copy link
Owner

Thank you very much @bnavigator for reporting this problem.

Just a quick question, if I change to dist_name = "validate_pyproject", will it solve the problem?

@bnavigator
Copy link
Author

I would have expected so, but it doesn't. My assumption was wrong. There is a difference between the .egg-info and .dist-info. The - vs. _ is irrelevant however:

Dist Info:

>>> import importlib.metadata as _M
>>> from pprint import pprint
>>> pprint(_M.files("validate_pyproject"))
[PackagePath('../../../bin/validate-pyproject'),
 PackagePath('validate_pyproject-0.9.dist-info/INSTALLER'),
 PackagePath('validate_pyproject-0.9.dist-info/LICENSE.txt'),
 PackagePath('validate_pyproject-0.9.dist-info/METADATA'),
 PackagePath('validate_pyproject-0.9.dist-info/RECORD'),
 PackagePath('validate_pyproject-0.9.dist-info/REQUESTED'),
 PackagePath('validate_pyproject-0.9.dist-info/WHEEL'),
 PackagePath('validate_pyproject-0.9.dist-info/direct_url.json'),
 PackagePath('validate_pyproject-0.9.dist-info/entry_points.txt'),
 PackagePath('validate_pyproject-0.9.dist-info/top_level.txt'),
 PackagePath('validate_pyproject/__init__.py'),
 PackagePath('validate_pyproject/__main__.py'),
 PackagePath('validate_pyproject/_vendor/__init__.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/LICENSE'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/__init__.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/__main__.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/draft04.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/draft06.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/draft07.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/exceptions.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/generator.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/indent.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/ref_resolver.py'),
 PackagePath('validate_pyproject/_vendor/fastjsonschema/version.py'),
 PackagePath('validate_pyproject/_vendor/vendoring_instructions.rst'),
 PackagePath('validate_pyproject/api.py'),
 PackagePath('validate_pyproject/cli.py'),
 PackagePath('validate_pyproject/error_reporting.py'),
 PackagePath('validate_pyproject/errors.py'),
 PackagePath('validate_pyproject/extra_validations.py'),
 PackagePath('validate_pyproject/formats.py'),
 PackagePath('validate_pyproject/plugins/__init__.py'),
 PackagePath('validate_pyproject/plugins/distutils.schema.json'),
 PackagePath('validate_pyproject/plugins/setuptools.schema.json'),
 PackagePath('validate_pyproject/pre_compile/NOTICE.template'),
 PackagePath('validate_pyproject/pre_compile/__init__.py'),
 PackagePath('validate_pyproject/pre_compile/__main__.py'),
 PackagePath('validate_pyproject/pre_compile/api-notice.template'),
 PackagePath('validate_pyproject/pre_compile/cli-notice.template'),
 PackagePath('validate_pyproject/pre_compile/cli.py'),
 PackagePath('validate_pyproject/pre_compile/main_file.template'),
 PackagePath('validate_pyproject/project_metadata.schema.json'),
 PackagePath('validate_pyproject/py.typed'),
 PackagePath('validate_pyproject/pyproject_toml.schema.json'),
 PackagePath('validate_pyproject/types.py'),
 PackagePath('validate_pyproject/vendoring/__init__.py'),
 PackagePath('validate_pyproject/vendoring/__main__.py'),
 PackagePath('validate_pyproject/vendoring/cli.py')]

egg-info:

>>> pprint(_M.files("validate_pyproject"))  
[PackagePath('.cirrus.yml'),
 PackagePath('.coveragerc'),
 PackagePath('.gitignore'),
 PackagePath('.isort.cfg'),
 PackagePath('.pre-commit-config.yaml'),
 PackagePath('.pre-commit-hooks.yaml'),
 PackagePath('.projections.json'),
 PackagePath('.readthedocs.yml'),
 PackagePath('AUTHORS.rst'),
 PackagePath('CHANGELOG.rst'),
 PackagePath('CONTRIBUTING.rst'),
 PackagePath('LICENSE.txt'),
 PackagePath('NOTICE.txt'),
 PackagePath('README.rst'),
 PackagePath('pyproject.toml'),
 PackagePath('setup.cfg'),
 PackagePath('setup.py'),
 PackagePath('tox.ini'),
 PackagePath('.github/workflows/ci.yml'),
 PackagePath('docs/Makefile'),
 PackagePath('docs/authors.rst'),
 PackagePath('docs/changelog.rst'),
 PackagePath('docs/conf.py'),
 PackagePath('docs/contributing.rst'),
 PackagePath('docs/dev-guide.rst'),
 PackagePath('docs/embedding.rst'),
 PackagePath('docs/faq.rst'),
 PackagePath('docs/index.rst'),
 PackagePath('docs/json-schemas.rst'),
 PackagePath('docs/license.rst'),
 PackagePath('docs/readme.rst'),
 PackagePath('docs/requirements.txt'),
 PackagePath('docs/schemas.rst'),
 PackagePath('docs/_static/.gitignore'),
 PackagePath('docs/_static/custom-adjustments.css'),
 PackagePath('src/validate_pyproject/__init__.py'),
 PackagePath('src/validate_pyproject/__main__.py'),
 PackagePath('src/validate_pyproject/api.py'),
 PackagePath('src/validate_pyproject/cli.py'),
 PackagePath('src/validate_pyproject/error_reporting.py'),
 PackagePath('src/validate_pyproject/errors.py'),
 PackagePath('src/validate_pyproject/extra_validations.py'),
 PackagePath('src/validate_pyproject/formats.py'),
 PackagePath('src/validate_pyproject/project_metadata.schema.json'),
 PackagePath('src/validate_pyproject/py.typed'),
 PackagePath('src/validate_pyproject/pyproject_toml.schema.json'),
 PackagePath('src/validate_pyproject/types.py'),
 PackagePath('src/validate_pyproject.egg-info/PKG-INFO'),
 PackagePath('src/validate_pyproject.egg-info/SOURCES.txt'),
 PackagePath('src/validate_pyproject.egg-info/dependency_links.txt'),
 PackagePath('src/validate_pyproject.egg-info/entry_points.txt'),
 PackagePath('src/validate_pyproject.egg-info/not-zip-safe'),
 PackagePath('src/validate_pyproject.egg-info/requires.txt'),
 PackagePath('src/validate_pyproject.egg-info/top_level.txt'),
 PackagePath('src/validate_pyproject/_vendor/__init__.py'),
 PackagePath('src/validate_pyproject/_vendor/vendoring_instructions.rst'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/LICENSE'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/__init__.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/__main__.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft04.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft06.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft07.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/exceptions.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/generator.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/indent.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/ref_resolver.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/version.py'),
 PackagePath('src/validate_pyproject/plugins/__init__.py'),
 PackagePath('src/validate_pyproject/plugins/distutils.schema.json'),
 PackagePath('src/validate_pyproject/plugins/setuptools.schema.json'),
 PackagePath('src/validate_pyproject/pre_compile/NOTICE.template'),
 PackagePath('src/validate_pyproject/pre_compile/__init__.py'),
 PackagePath('src/validate_pyproject/pre_compile/__main__.py'),
 PackagePath('src/validate_pyproject/pre_compile/api-notice.template'),
 PackagePath('src/validate_pyproject/pre_compile/cli-notice.template'),
 PackagePath('src/validate_pyproject/pre_compile/cli.py'),
 PackagePath('src/validate_pyproject/pre_compile/main_file.template'),
 PackagePath('src/validate_pyproject/vendoring/__init__.py'),
 PackagePath('src/validate_pyproject/vendoring/__main__.py'),
 PackagePath('src/validate_pyproject/vendoring/cli.py'),
 PackagePath('tests/__init__.py'),
 PackagePath('tests/conftest.py'),
 PackagePath('tests/helpers.py'),
 PackagePath('tests/test_api.py'),
 PackagePath('tests/test_cli.py'),
 PackagePath('tests/test_error_reporting.py'),
 PackagePath('tests/test_examples.py'),
 PackagePath('tests/test_formats.py'),
 PackagePath('tests/test_json_schema_summary.py'),
 PackagePath('tests/test_plugins.py'),
 PackagePath('tests/test_pre_compile.py'),
 PackagePath('tests/test_vendoring.py'),
 PackagePath('tests/examples/atoml/LICENSE'),
 PackagePath('tests/examples/atoml/pyproject.toml'),
 PackagePath('tests/examples/flit/LICENSE'),
 PackagePath('tests/examples/flit/pyproject.toml'),
 PackagePath('tests/examples/pdm/LICENSE'),
 PackagePath('tests/examples/pdm/pyproject.toml'),
 PackagePath('tests/examples/pep_text/pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/01-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/02-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/03-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/04-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/05-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/06-pyproject.toml'),
 PackagePath('tests/examples/pretend-setuptools/07-pyproject.toml'),
 PackagePath('tests/examples/simple/dynamic-version.toml'),
 PackagePath('tests/examples/simple/minimal.toml'),
 PackagePath('tests/examples/trampolim/LICENSE'),
 PackagePath('tests/examples/trampolim/pyproject.toml'),
 PackagePath('tests/invalid-examples/pdm/LICENSE'),
 PackagePath('tests/invalid-examples/pdm/invalid-version/errors.txt'),
 PackagePath('tests/invalid-examples/pdm/invalid-version/pyproject.toml'),
 PackagePath('tests/invalid-examples/pdm/redefining-as-dynamic/errors.txt'),
 PackagePath('tests/invalid-examples/pdm/redefining-as-dynamic/pyproject.toml'),
 PackagePath('tests/invalid-examples/pep621/missing-fields/missing-version-with-dynamic.errors.txt'),
 PackagePath('tests/invalid-examples/pep621/missing-fields/missing-version-with-dynamic.toml'),
 PackagePath('tests/invalid-examples/pep621/missing-fields/missing-version.errors.txt'),
 PackagePath('tests/invalid-examples/pep621/missing-fields/missing-version.toml'),
 PackagePath('tests/invalid-examples/pep621/non-standardised-project-fields/author_instead_of_authors.errors.txt'),
 PackagePath('tests/invalid-examples/pep621/non-standardised-project-fields/author_instead_of_authors.toml'),
 PackagePath('tests/invalid-examples/pep621/non-standardised-project-fields/requires_instead_of_dependencies.errors.txt'),
 PackagePath('tests/invalid-examples/pep621/non-standardised-project-fields/requires_instead_of_dependencies.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/cmdclass/invalid-value.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/cmdclass/invalid-value.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/dependencies/invalid-extra-name.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/dependencies/invalid-extra-name.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/package-dir/invalid-name.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/package-dir/invalid-name.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/packages/missing-find-arguments.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/packages/missing-find-arguments.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/license/both-text-and-file.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/license/both-text-and-file.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/license/empty.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/license/empty.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/readme/readme-as-array.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/readme/readme-as-array.toml'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/readme/readme-without-content-type.errors.txt'),
 PackagePath('tests/invalid-examples/pretend-setuptools/pep621/readme/readme-without-content-type.toml'),
 PackagePath('tests/json_schema_summary/array-contains.example'),
 PackagePath('tests/json_schema_summary/array-no-items.example'),
 PackagePath('tests/json_schema_summary/array-prefix-items.example'),
 PackagePath('tests/json_schema_summary/array-simple.example'),
 PackagePath('tests/json_schema_summary/if-then-else.example'),
 PackagePath('tests/json_schema_summary/if-then-else2.example'),
 PackagePath('tests/json_schema_summary/not.example'),
 PackagePath('tests/json_schema_summary/object-no-properties.example'),
 PackagePath('tests/json_schema_summary/object-pattern-properties.example'),
 PackagePath('tests/json_schema_summary/object-property-names.example'),
 PackagePath('tests/json_schema_summary/oneof.example'),
 PackagePath('tests/pre_compile/test_cli.py')]

Seems like importlib.metadata.files() returns the content of the .dist-info/RECORD or .egg-info/SOURCES.txt. And the latter is not the installed files.

@bnavigator
Copy link
Author

https://docs.python.org/3/library/importlib.metadata.html#distribution-versions and https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html#sources-txt-source-files-manifest seem to contradict each other here.

In line with #50 it might be worth not to rely on importlib.metadata.files() at all.

@bnavigator bnavigator mentioned this issue Jul 14, 2022
Closed
@bnavigator
Copy link
Author

Update: With importlib_metadata 4.11.4 overriding the importlib.metadata from Python 3.10.5, the filelist changes, but ist still wrong:

abuild@skylab:~> python3
Python 3.10.5 (main, Jun 06 2022, 22:34:44) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib_metadata as _M
>>> from pprint import pprint
>>> pprint(_M.files("validate_pyproject"))
[PackagePath('LICENSE.txt'),
 PackagePath('README.rst'),
 PackagePath('pyproject.toml'),
 PackagePath('setup.cfg'),
 PackagePath('setup.py'),
 PackagePath('src/validate_pyproject/__init__.py'),
 PackagePath('src/validate_pyproject/__main__.py'),
 PackagePath('src/validate_pyproject/api.py'),
 PackagePath('src/validate_pyproject/cli.py'),
 PackagePath('src/validate_pyproject/error_reporting.py'),
 PackagePath('src/validate_pyproject/errors.py'),
 PackagePath('src/validate_pyproject/extra_validations.py'),
 PackagePath('src/validate_pyproject/formats.py'),
 PackagePath('src/validate_pyproject/types.py'),
 PackagePath('src/validate_pyproject.egg-info/PKG-INFO'),
 PackagePath('src/validate_pyproject.egg-info/SOURCES.txt'),
 PackagePath('src/validate_pyproject.egg-info/dependency_links.txt'),
 PackagePath('src/validate_pyproject.egg-info/entry_points.txt'),
 PackagePath('src/validate_pyproject.egg-info/not-zip-safe'),
 PackagePath('src/validate_pyproject.egg-info/requires.txt'),
 PackagePath('src/validate_pyproject.egg-info/top_level.txt'),
 PackagePath('src/validate_pyproject/_vendor/__init__.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/__init__.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/__main__.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft04.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft06.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/draft07.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/exceptions.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/generator.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/indent.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/ref_resolver.py'),
 PackagePath('src/validate_pyproject/_vendor/fastjsonschema/version.py'),
 PackagePath('src/validate_pyproject/plugins/__init__.py'),
 PackagePath('src/validate_pyproject/pre_compile/__init__.py'),
 PackagePath('src/validate_pyproject/pre_compile/__main__.py'),
 PackagePath('src/validate_pyproject/pre_compile/cli.py'),
 PackagePath('src/validate_pyproject/vendoring/__init__.py'),
 PackagePath('src/validate_pyproject/vendoring/__main__.py'),
 PackagePath('src/validate_pyproject/vendoring/cli.py')]
>>>

@jaraco
Copy link

jaraco commented Jul 14, 2022

Ideally, Setuptools would not generate egg-info, and would instead generate dist-info with a RECORD. If it did that, then importlib.metadata.files would provide a consistent experience. The fallback to SOURCES.txt for egg-info is there to give its best effort at the list of files in the project, but as you can see, it's inconsistent.

@bnavigator
Copy link
Author

I wish we would live in an ideal world 😆

@abravalheri
Copy link
Owner

abravalheri commented Jul 14, 2022 •
edited
Loading

@bnavigator what is the motivation behind using the deprecated python setup.py install? (I am imagining that you are using install...).

It should be possible to support this use case, however there will be an increase in complexity and maintenance effort, which I would like to avoid...

It non trivial to satisfy these constraints simultaneously: a) my desire to avoid double bookkeeping, b) (potential) lack of symlinks on Windows, c) Legacy installation methods like python setup.py, d) Github discovery/gui of/for license files.

@bnavigator
Copy link
Author

bnavigator commented Jul 14, 2022 •
edited
Loading

No motivation at all. It's the legacy way of packaging and so far the default when generating recipes for a rpm build. I switched to PEP517 and made the error go away.

The easiest way to avoid future errors like this is to remove the setup.py completely and force your users to use PEP517.

Ironically I only stumbled over this because I am updating the setuptools package itself and validate-pyproject is in the dependency graph for the test suite.

@abravalheri
Copy link
Owner

The easiest way to avoid future errors like this is to remove the setup.py completely and force your users to use PEP517.

Thank you very much for the information @bnavigator. I think I will follow this direction.

abravalheri added a commit that referenced this issue Sep 1, 2022
@abravalheri
Remove setup.py
1658ea6 
With PEP 660 implemented in setuptools, this should be safe.

Closes #52.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jaraco @abravalheri @bnavigator