Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild #2416

andrew-m-leonard · 2021-01-28T13:43:15Z

Add the facility to clean the build workspace or just the workspace/build/src/build and workspace/target binary output at the end of the build. This provides a large saving of disk space, up to 30Gb for Build Output alone and an additional 6Gb for the whole workspace. The default settings for generated pipelines are:

cleanWorkspaceAfterBuild : false
cleanWorkspaceBuildOutputAfterBuild : true
Additionally, cleanWorkspaceAfterBuild can be specified in the jdk platform configuration if a specific platform requires cleaning only.

Signed-off-by: Andrew Leonard [email protected]

AdamBrousseau · 2021-01-29T00:17:21Z

I believe this pr addresses #2296

pipelines/build/common/openjdk_build_pipeline.groovy

sxa

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

andrew-m-leonard · 2021-01-29T19:28:09Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

It's any file that has ".." in it, the 2 i've discovered so far:

17:51:00  workspace/build/src/build/windows-x86_64-normal-clientANDserver-release/support/gensrc/java.base/java/nio/_the...marker
17:31:58  workspace/build/src/build/windows-x86_64-normal-server-release/jdk/gensrc/java/nio/_the..

andrew-m-leonard · 2021-01-29T19:29:05Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

Yeah, I thought the print was useful debug for the moment, but i'll remove

andrew-m-leonard · 2021-01-29T19:30:40Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

Yes, I like the -type f, good idea

andrew-m-leonard · 2021-01-29T19:32:25Z

@sxa The other solution is not use CleanWs, just use rm -rf on the specific folders?
The logic behind CleanWs using DirectoryScanner is just overkill, especially for what we're doing

sxa · 2021-01-30T13:14:51Z

@sxa The other solution is not use CleanWs, just use rm -rf on the specific folders?

i'm thinking that's probably the best option - keeps it simple, but if we do that let's add a comment pointing to the issue as an explanation of why we're not using cleanWs

andrew-m-leonard · 2021-02-01T17:04:37Z

run tests

adoptopenjdk-github-bot · 2021-02-01T17:07:43Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-01T20:09:09Z

run tests

adoptopenjdk-github-bot · 2021-02-01T20:09:37Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-01T20:23:27Z

run tests

sxa

Looking good now but I won't formally approve til I have another look over it in the morning (unless two other people want to approve in the meantime!)

Would be good to check a few random machines to see how much space workspace/build-scripts is taking and compare it to the same numbers a week after merging this when it should have taken effect across most of the machines. Should particularly help the scaleway arm32 build systems

Clicked wrong button - as per previous comment not doing full approval til tomorrow :-)

adoptopenjdk-github-bot · 2021-02-01T22:05:19Z

🟢 PR TESTER RESULT 🟢

✅ All pipelines passed! ✅

sxa

Other than the naming which I think it's now confusing I think this is ready

pipelines/build/common/build_base_file.groovy

Signed-off-by: Andrew Leonard <[email protected]>

andrew-m-leonard · 2021-02-02T12:27:41Z

run tests

sxa

Yep I'll take that - thanks. Clear enough now I reckon :-)

sxa · 2021-02-02T15:23:43Z

ping @karianna as you are currently blocking this

adoptopenjdk-github-bot · 2021-02-02T16:10:30Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-02T16:29:51Z

run tests

adoptopenjdk-github-bot · 2021-02-02T20:12:49Z

🟢 PR TESTER RESULT 🟢

✅ All pipelines passed! ✅

AdamBrousseau · 2021-02-05T08:45:19Z

pipelines/build/common/openjdk_build_pipeline.groovy

-                                context.cleanWs notFailBuild: true, disableDeferredWipeout: true, deleteDirs: true
+                            // Note: Underlying org.apache DirectoryScanner used by cleanWs has a bug scanning where it misses files containing ".." so use rm -rf instead
+                            // Issue: https://issues.jenkins.io/browse/JENKINS-64779
+                            if (context.WORKSPACE != null && !context.WORKSPACE.isEmpty()) {


i think if (context.WORKSPACE) achieves the same thing.

AdamBrousseau · 2021-02-05T08:45:58Z

pipelines/build/common/openjdk_build_pipeline.groovy

+                            // Issue: https://issues.jenkins.io/browse/JENKINS-64779
+                            if (context.WORKSPACE != null && !context.WORKSPACE.isEmpty()) {
+                                context.println "Cleaning workspace files: " + context.WORKSPACE + "/*"
+                                context.sh(script: "rm -rf " + context.WORKSPACE + "/*")


I think this ignores the . hidden directories (eg. .git folder). Which is causing me issues on my win boxes.

andrew-m-leonard marked this pull request as draft January 28, 2021 13:43

M-Davies added enhancement Issues that enhance the code or documentation of the repo in any way jenkins Issues that enhance or fix our jenkins server labels Jan 28, 2021

andrew-m-leonard force-pushed the clearbuild branch 5 times, most recently from f8b6893 to 6b15fe6 Compare January 28, 2021 20:13

andrew-m-leonard force-pushed the clearbuild branch 7 times, most recently from a1bceee to 55e068d Compare January 29, 2021 15:48

andrew-m-leonard marked this pull request as ready for review January 29, 2021 18:43

sxa reviewed Jan 29, 2021

View reviewed changes

pipelines/build/common/openjdk_build_pipeline.groovy Outdated Show resolved Hide resolved

sxa reviewed Jan 29, 2021

View reviewed changes

sxa mentioned this pull request Jan 31, 2021

pipelines: Use variables to remove Windows workspaces #2410

Closed

andrew-m-leonard force-pushed the clearbuild branch 4 times, most recently from 7c60e39 to e0242a8 Compare February 1, 2021 09:47

andrew-m-leonard force-pushed the clearbuild branch from 5f5cb31 to 29714a6 Compare February 1, 2021 17:02

andrew-m-leonard requested a review from karianna February 1, 2021 17:04

M-Davies mentioned this pull request Feb 1, 2021

CleanWs at end of build should be runtime configurable #2296

Closed

M-Davies linked an issue Feb 1, 2021 that may be closed by this pull request

CleanWs at end of build should be runtime configurable #2296

Closed

sxa reviewed Feb 1, 2021

View reviewed changes

sxa previously approved these changes Feb 1, 2021

View reviewed changes

sxa reviewed Feb 2, 2021

View reviewed changes

pipelines/build/common/build_base_file.groovy Outdated Show resolved Hide resolved

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild

50bf26e

Signed-off-by: Andrew Leonard <[email protected]>

andrew-m-leonard force-pushed the clearbuild branch from 29714a6 to 50bf26e Compare February 2, 2021 12:27

sxa approved these changes Feb 2, 2021

View reviewed changes

karianna approved these changes Feb 2, 2021

View reviewed changes

karianna merged commit 96068f0 into adoptium:master Feb 2, 2021

Willsparker mentioned this pull request Feb 3, 2021

pipelines: Find a better solution for deleting the _the.. after Windows builds #2205

Closed

AdamBrousseau reviewed Feb 5, 2021

View reviewed changes

sxa assigned andrew-m-leonard Feb 8, 2021

sxa mentioned this pull request Feb 8, 2021

Machines low on disk space: zLinux build machines adoptium/infrastructure#1652

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild #2416

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild #2416

andrew-m-leonard commented Jan 28, 2021 •

edited

Loading

AdamBrousseau commented Jan 29, 2021 •

edited

Loading

sxa left a comment •

edited

Loading

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

sxa commented Jan 30, 2021

andrew-m-leonard commented Feb 1, 2021

adoptopenjdk-github-bot commented Feb 1, 2021

andrew-m-leonard commented Feb 1, 2021

adoptopenjdk-github-bot commented Feb 1, 2021

andrew-m-leonard commented Feb 1, 2021

sxa left a comment

adoptopenjdk-github-bot commented Feb 1, 2021

sxa left a comment

andrew-m-leonard commented Feb 2, 2021

sxa left a comment

sxa commented Feb 2, 2021

adoptopenjdk-github-bot commented Feb 2, 2021

andrew-m-leonard commented Feb 2, 2021

adoptopenjdk-github-bot commented Feb 2, 2021

AdamBrousseau Feb 5, 2021

AdamBrousseau Feb 5, 2021 •

edited

Loading

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild #2416

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild #2416

Conversation

andrew-m-leonard commented Jan 28, 2021 • edited Loading

AdamBrousseau commented Jan 29, 2021 • edited Loading

sxa left a comment • edited Loading

Choose a reason for hiding this comment

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

andrew-m-leonard commented Jan 29, 2021

sxa commented Jan 30, 2021

andrew-m-leonard commented Feb 1, 2021

adoptopenjdk-github-bot commented Feb 1, 2021

🟠 PR TESTER RESULT 🟠

andrew-m-leonard commented Feb 1, 2021

adoptopenjdk-github-bot commented Feb 1, 2021

🟠 PR TESTER RESULT 🟠

andrew-m-leonard commented Feb 1, 2021

sxa left a comment

Choose a reason for hiding this comment

adoptopenjdk-github-bot commented Feb 1, 2021

🟢 PR TESTER RESULT 🟢

sxa left a comment

Choose a reason for hiding this comment

andrew-m-leonard commented Feb 2, 2021

sxa left a comment

Choose a reason for hiding this comment

sxa commented Feb 2, 2021

adoptopenjdk-github-bot commented Feb 2, 2021

🟠 PR TESTER RESULT 🟠

andrew-m-leonard commented Feb 2, 2021

adoptopenjdk-github-bot commented Feb 2, 2021

🟢 PR TESTER RESULT 🟢

AdamBrousseau Feb 5, 2021

Choose a reason for hiding this comment

AdamBrousseau Feb 5, 2021 • edited Loading

Choose a reason for hiding this comment

andrew-m-leonard commented Jan 28, 2021 •

edited

Loading

AdamBrousseau commented Jan 29, 2021 •

edited

Loading

sxa left a comment •

edited

Loading

AdamBrousseau Feb 5, 2021 •

edited

Loading