Skip to content

Commit

Permalink
[SPARK-32556][INFRA] Fix release script to have urlencoded passwords …
Browse files Browse the repository at this point in the history
…where required

### What changes were proposed in this pull request?
1. URL encode the `ASF_PASSWORD` of the release manager.
2. Update the image to install `qpdf` and `jq` dep
3. Increase the JVM HEAM memory for maven build.

### Why are the changes needed?
Release script takes hours to run, and if a single failure happens about somewhere midway, then either one has to get down to manually doing stuff or re run the entire script. (This is my understanding) So, I have made the fixes of a few failures, discovered so far.

1. If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step.
`git clone "https://$ASF_USERNAME:$ASF_PASSWORD$ASF_SPARK_REPO" -b $GIT_BRANCH`

          ^^^ Fails with bad URL

`ASF_USERNAME` may not be URL encoded, but we need to encode `ASF_PASSWORD`.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
By running the release for branch-2.4, using both type of passwords, i.e. passwords with special chars and without it.

Closes #29373 from ScrapCodes/release-script-fix2.

Lead-authored-by: Prashant Sharma <[email protected]>
Co-authored-by: Prashant Sharma <[email protected]>
Signed-off-by: Prashant Sharma <[email protected]>
  • Loading branch information
ScrapCodes and ScrapCodes committed Aug 7, 2020
1 parent d5682c1 commit 6c3d0a4
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 2 deletions.
6 changes: 5 additions & 1 deletion dev/create-release/release-tag.sh
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,12 @@ init_maven_sbt

ASF_SPARK_REPO="gitbox.apache.org/repos/asf/spark.git"

function uriencode { jq -nSRr --arg v "$1" '$v|@uri'; }

declare -r ENCODED_ASF_PASSWORD=$(uriencode "$ASF_PASSWORD")

rm -rf spark
git clone "https://$ASF_USERNAME:$ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
git clone "https://$ASF_USERNAME:$ENCODED_ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
cd spark

git config user.name "$GIT_NAME"
Expand Down
2 changes: 1 addition & 1 deletion dev/create-release/release-util.sh
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ function init_maven_sbt {
if [[ $JAVA_VERSION < "1.8." ]]; then
# Needed for maven central when using Java 7.
SBT_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
MVN_EXTRA_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
MVN_EXTRA_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dhttps.protocols=TLSv1.1,TLSv1.2"
MVN="$MVN $MVN_EXTRA_OPTS"
fi
export MVN MVN_EXTRA_OPTS SBT_OPTS
Expand Down
2 changes: 2 additions & 0 deletions dev/create-release/spark-rm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
$APT_INSTALL nodejs && \
# Install needed python packages. Use pip for installing packages (for consistency).
$APT_INSTALL libpython3-dev python3-pip python3-setuptools && \
# qpdf is required for CRAN checks to pass.
$APT_INSTALL qpdf jq && \
# Change default python version to python3.
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \
Expand Down

0 comments on commit 6c3d0a4

Please sign in to comment.