Dependencies: refactor with rules_python update #14329
Conversation
…les_python update. Signed-off-by: Michael Payne <[email protected]>
repokitteh-read-only
bot
added
the
deps
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
… fix release_dates.py. Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
|
/retest |
|
Retrying Azure Pipelines: |
…ison resilience. Signed-off-by: Michael Payne <[email protected]>
| for tag in tags.reversed: | ||
| if tag.name == github_release.version: | ||
| current_metadata_tag_commit_date = tag.commit.commit.committer.date | ||
| # Exclude pre-releases like release candidates (eg. apache/kafka) |
There was a problem hiding this comment.
Something tells me that folks will have ways of creating tag names that are going to break us, but I'm happy to land what you have and iterate.
There was a problem hiding this comment.
We already have a bunch of different approaches here with leading v on versions, straight numbers, some with underscores, some with dashes, some with both underscores and dashes and some with alpha numeric strings etc. I think the version import in packaging will help with this and so far the scripts runs OK against our repositories. But definitely a potential source of fragility.
There was a problem hiding this comment.
Bigger issue here is a Windows CI failure with the new rules_python 0.1.0 approach - https://github.com/bazelbuild/rules_python/releases/tag/0.1.0. Fails consistently trying to build the Thrift python bits - https://dev.azure.com/cncf/envoy/_build/results?buildId=60601&view=logs&j=b840a642-5ff3-5357-2e4b-e06e40b0cffd&t=67965174-5100-5631-9dc0-68b9f0aacb53. Any ideas @envoyproxy/windows-dev ?
There was a problem hiding this comment.
Honest answer is that I dont know why/how this error is coming from. There are multiple mentions across the internet and it seems that this helps other folks.
There was a problem hiding this comment.
I think rules_python might have to update its dependency on setuptools, it is pretty old and looks like the version in use has some incorrect logic around finding the vcruntime*.dll
There was a problem hiding this comment.
There was a problem hiding this comment.
Specifically in https://github.com/pypa/setuptools/blob/v42.0.2/setuptools/msvc.py#L1540-L1572
I think https://github.com/pypa/setuptools/blob/v42.0.2/setuptools/msvc.py#L1564-L1566 might be wrong
Looking at a build container these are all the paths I see vcruntime*.dll installed:
/c/build/tmp/external/remotejdk11_win/bin/vcruntime140.dll
/c/build/tmp/_bazel_ContainerAdministrator/install/7b45c91da117b160cc3b12126324930f/embedded_tools/jdk/bin/vcruntime140.dll
/c/Program Files/LLVM/bin/vcruntime140.dll
/c/Program Files/LLVM/bin/vcruntime140_1.dll
/c/Program Files/Python38/vcruntime140.dll
/c/Program Files/Python38/vcruntime140_1.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140d.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140_1d.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140d.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140_1d.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/x64/Microsoft.VC142.CRT/vcruntime140.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/x64/Microsoft.VC142.CRT/vcruntime140_1.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime140.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime140_1.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/vcruntime140.dll
/c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/vcruntime140_1.dll
/c/Windows/System32/vcruntime140.dll
/c/Windows/System32/vcruntime140d.dll
/c/Windows/System32/vcruntime140_1.dll
/c/Windows/System32/vcruntime140_1d.dll
/c/Windows/SysWOW64/vcruntime140.dll
/c/Windows/SysWOW64/vcruntime140d.dll
I think setuptools is looking for /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime*.dll but the problematic part is the Microsoft.VC142.CRT rather than Microsoft.VC140.CRT in the path, which is incorrect for the latest versions of MSVC/VCRedist (or inconsistency from Microsoft, idk)
There was a problem hiding this comment.
The logic around finding vcruntime*.dll has completely changed so might be worth an update: https://github.com/pypa/setuptools/blob/v51.1.0/setuptools/msvc.py#L217, looks like it handles that path better
There was a problem hiding this comment.
This diff seems to have fixed things:
diff --git a/bazel/repositories.bzl b/bazel/repositories.bzl
index 92c2b0e84..f4ed59589 100644
--- a/bazel/repositories.bzl
+++ b/bazel/repositories.bzl
@@ -562,7 +562,11 @@ def _com_google_absl():
)
def _com_google_protobuf():
- external_http_archive("rules_python")
+ external_http_archive(
+ name = "rules_python",
+ patches = ["@envoy//bazel:rules_python.patch"],
+ patch_args = ["-p1"],
+ )
external_http_archive(
"com_google_protobuf",
patches = ["@envoy//bazel:protobuf.patch"],
diff --git a/bazel/rules_python.patch b/bazel/rules_python.patch
new file mode 100644
index 000000000..81929031d
--- /dev/null
+++ b/bazel/rules_python.patch
@@ -0,0 +1,15 @@
+diff --git a/python/pip_install/repositories.bzl b/python/pip_install/repositories.bzl
+index df63674..80824e4 100644
+--- a/python/pip_install/repositories.bzl
++++ b/python/pip_install/repositories.bzl
+@@ -16,8 +16,8 @@ _RULE_DEPS = [
+ ),
+ (
+ "pypi__setuptools",
+- "https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl",
+- "c8abd0f3574bc23afd2f6fd2c415ba7d9e097c8a99b845473b0d957ba1e2dac6",
++ "https://files.pythonhosted.org/packages/3d/f2/1489d3b6c72d68bf79cd0fba6b6c7497df4ebf7d40970e2d7eceb8d0ea9c/setuptools-51.0.0-py3-none-any.whl",
++ "8c177936215945c9a37ef809ada0fab365191952f7a123618432bbfac353c529",
+ ),
+ (
+ "pypi__wheel",
Signed-off-by: Michael Payne <[email protected]>
|
LGTM @moderation can you back out the problematic |
Signed-off-by: Michael Payne <[email protected]>
Signed-off-by: Michael Payne <[email protected]>
|
Taking a closer look now, trying to repro at my devbox @moderation |
| @@ -211,6 +211,7 @@ build:remote --auth_enabled=true | |||
| build:remote --remote_download_toplevel | |||
|
|
|||
| # Windows bazel does not allow sandboxed as a spawn strategy | |||
| build:remote-windows --action_env=WRAPT_INSTALL_EXTENSIONS=false | |||
There was a problem hiding this comment.
Probably don't need this and the other change to this file
|
This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions! |
github-actions
bot
added
the
stale
|
This pull request has been automatically closed because it has not had activity in the last 37 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions! |
Commit Message: Dependencies: refactor to remove bazel/repositories_extra.bzl with rules_python update
Additional Description:
rules_python0.1.0(release notes). Move pip imports frombazel/repositories_extra.bzland consolidate intobazel/dependency_imports.bzlsixfrom pythonhosted to Github location so we can verify release datebazel/repository_locations.bzlso thatrelease_datefollowsversionandsha256to aid with dependency maintainancetools/dependency/release_dates.pyto check for release metadata vs. current approach of checking for latest commit in a release which may not equal release date. Makes additional calls to the Github API and not super efficient but works. /cc @htuch for Python review.Risk Level: Low
Testing:
bazel --nohome_rc test //test/...,bazel --nohome_rc test @envoy_api_canonical//test/... @envoy_api_canonical//tools/...,bazel --nohome_rc build @envoy_api_canonical//envoy/...,tools/dependency/release_dates.py bazel/repository_locations.bzlDocs Changes: None required
Release Notes: None required
Signed-off-by: Michael Payne [email protected]