Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Defined Shims Should Be Declared In POM #10167

Merged
merged 5 commits into from
Jan 10, 2024
Merged

Defined Shims Should Be Declared In POM #10167

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6cf9b61
check shim exists in all.buildversions
razajafri Jan 9, 2024
b673fba
Signing off
razajafri Jan 9, 2024
12d9da7
addressed review comments
razajafri Jan 9, 2024
7f2165d
Merge remote-tracking branch 'origin/branch-24.02' into undeclared-shim
razajafri Jan 10, 2024
6b9a29d
return the shim version from the map method
razajafri Jan 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion build/shimplify.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,10 @@ def __generate_symlinks():
path,
build_ver_arr))

def __map_version_array(shim_json_string):
shim_ver = str(json.loads(shim_json_string).get('spark'))
assert shim_ver in __all_shims_arr, "all.buildvers in pom.xml does not contain %s" % shim_ver
return shim_ver

def __traverse_source_tree_of_all_shims(src_type, func):
"""Walks src/<src_type>/sparkXYZ"""
Expand All @@ -390,7 +394,7 @@ def __traverse_source_tree_of_all_shims(src_type, func):
shim_arr = shim_match.group(1).split(os.linesep)
assert len(shim_arr) > 0, "invalid empty shim comment,"\
"orphan shim files should be deleted"
build_ver_arr = map(lambda s: str(json.loads(s).get('spark')), shim_arr)
build_ver_arr = map(__map_version_array, shim_arr)
__log.debug("extracted shims %s", build_ver_arr)
assert build_ver_arr == sorted(build_ver_arr),\
"%s shim list is not properly sorted" % shim_file_path
Expand Down