Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Pex to 2.1.95. #16110

Merged
merged 4 commits into from
Jul 10, 2022
Merged

Upgrade Pex to 2.1.95. #16110

merged 4 commits into from
Jul 10, 2022

Conversation

jsirois
Copy link
Contributor

@jsirois jsirois commented Jul 9, 2022

This picks up several fixes that make PEX environments more robust to
launch and use. It also picks up a new --target-system feature for
universal resolves which is used in this change to limit our locks to
Linux and Mac to pre-emptively avoid otherwise spurious lock errors.

See the changelog here:
https://github.com/pantsbuild/pex/releases/tag/v2.1.95

[ci skip-rust]
[ci skip-build-wheels]

This picks up several fixes that make PEX environments more robust to
launch and use. It also picks up a new `--target-system` feature for
universal resolves which is used in this change to limit our locks to
Linux and Mac to pre-emptively avoid otherwise spurious lock errors.

ALl Python lock files are re-generated with the new `--target-system`
defaulting in place in order to drop a bunch of Windows-only
dependencies and artifacts.

See the changelog here:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.95

[ci skip-rust]
[ci skip-build-wheels]
@jsirois
Copy link
Contributor Author

jsirois commented Jul 9, 2022

If folks think regenerating all Python lock files was too aggressive, speak up. I don't have a strong opinion here and could restrict to just Pex and Lambdex as is normal with the Pex upgrades.

@jsirois jsirois added this to the 2.13.x milestone Jul 9, 2022
@jsirois
Copy link
Contributor Author

jsirois commented Jul 9, 2022

The user reporting the issue --target-system solves is on Pants 2.11.0 but they have an acceptable workaround. I'd like to only pick this back to 2.13.x as a result, which is the only branch on 2.1.94. The rest are on 2.1.90 max which is a big leap.

"url": "https://files.pythonhosted.org/packages/55/8d/74a75635f2c3c914ab5b3850112fd4b0c8039975ecb320e4449aa363ba54/atomicwrites-1.4.0.tar.gz"
}
],
"project_name": "atomicwrites",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah! Apropos removal given recent events.

@@ -176,6 +176,18 @@ async def generate_lockfile(
"--style=universal",
"--resolver-version",
"pip-2020-resolver",
# PEX files currently only run on Linux and Mac machines; so we hard code this
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one code change in all this noise version bump / ./build-support/bin/generate_all_lockfiles.sh --all noise.

[ci skip-rust]
[ci skip-build-wheels]
@jsirois
Copy link
Contributor Author

jsirois commented Jul 9, 2022

The mypy-protobuf issue was: https://github.com/pantsbuild/pants/runs/7265712739?check_suite_focus=true#step:11:345

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

This was due to this change on PyPI:
  https://discuss.python.org/t/backwards-incompatible-change-to-pypi-json-api/17154

Which was fixed in Poetry yesterday here:
  python-poetry/poetry#5973

Unfortunately, the full fix requires blowing away the non-Pants
controlled Poetry cache at `~/.cache/pypoetry` on Linux and
`~/Library/Caches/pypoetry` on Mac.

Fixes pantsbuild#16111

[ci skip-rust]
[ci skip-build-wheels]
@jsirois
Copy link
Contributor Author

jsirois commented Jul 9, 2022

The Poetry issue is fixed over in #16112 which I've merged in. Interestingly, that requires no lockfile re-gen since Poetry appears to be the sole Python tool not using a lockfile! I'll not tackle that here.

Copy link
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize we had this many tool lockfiles!

@jsirois jsirois merged commit e7988a3 into pantsbuild:main Jul 10, 2022
@jsirois jsirois deleted the pex/upgrade branch July 10, 2022 01:54
jsirois added a commit to jsirois/pants that referenced this pull request Jul 10, 2022
This picks up several fixes that make PEX environments more robust to
launch and use. It also picks up a new `--target-system` feature for
universal resolves which is used in this change to limit our locks to
Linux and Mac to pre-emptively avoid otherwise spurious lock errors.

ALl Python lock files are re-generated with the new `--target-system`
defaulting in place in order to drop a bunch of Windows-only
dependencies and artifacts.

See the changelog here:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.95

(cherry picked from commit e7988a3)

[ci skip-rust]

[ci skip-build-wheels]
jsirois added a commit to jsirois/pants that referenced this pull request Jul 10, 2022
This picks up several fixes that make PEX environments more robust to
launch and use. It also picks up a new `--target-system` feature for
universal resolves which is used in this change to limit our locks to
Linux and Mac to pre-emptively avoid otherwise spurious lock errors.

ALl Python lock files are re-generated with the new `--target-system`
defaulting in place in order to drop a bunch of Windows-only
dependencies and artifacts.

See the changelog here:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.95

(cherry picked from commit e7988a3)

[ci skip-rust]
[ci skip-build-wheels]
@jsirois
Copy link
Contributor Author

jsirois commented Jul 10, 2022

Poetry tool lockfile: #16117

jsirois added a commit that referenced this pull request Jul 10, 2022
…-picks of #16110 & #15960) (#16116)

Cherry-pick Pex upgrade and also cherry-pick Black upgrade to avoid pulling in a version of Click that breaks older Black during lock regeneration.

---
Upgrade Pex to 2.1.95. (Cherry-pick of #16110)

This picks up several fixes that make PEX environments more robust to
launch and use. It also picks up a new `--target-system` feature for
universal resolves which is used in this change to limit our locks to
Linux and Mac to pre-emptively avoid otherwise spurious lock errors.

ALl Python lock files are re-generated with the new `--target-system`
defaulting in place in order to drop a bunch of Windows-only
dependencies and artifacts.

See the changelog here:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.95

(cherry picked from commit e7988a3)

---
Upgrade default version of black to 22.6.0 (Cherry-pick of #15960)
    
(cherry picked from commit ec1cf20)
benjyw added a commit to benjyw/pants that referenced this pull request Jul 11, 2022
This reverts commit e7988a3.

[ci skip-rust]

[ci skip-build-wheels]
benjyw added a commit that referenced this pull request Jul 11, 2022
This reverts #16110, as that changed caused wheel building to fail.

To observe this failure, run ./build-support/bin/release.sh build-wheels

Note that the revert was not clean due to #15951 also writing user_reqs.lock.
So I regenerated that file to ensure validity, and manually verified that the
failure above is fixed.

[ci skip-rust]

[ci skip-build-wheels]
benjyw added a commit to benjyw/pants that referenced this pull request Jul 11, 2022
Note that the revert didn't apply cleanly, so
I regenerated all the lockfiles.

[ci skip-rust]

[ci skip-build-wheels]
benjyw added a commit to benjyw/pants that referenced this pull request Jul 11, 2022
Note that the revert didn't apply cleanly, so
I regenerated all the lockfiles.

[ci skip-rust]

[ci skip-build-wheels]
benjyw added a commit to benjyw/pants that referenced this pull request Jul 11, 2022
… (Cherry-picks of pantsbuild#16110 & pantsbuild#15960) (pantsbuild#16116)"

This reverts commit 0987300.

[ci skip-rust]

[ci skip-build-wheels]
@stuhood stuhood mentioned this pull request Jul 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants