chore: create CI artifacts for DEV usage #21645

mdelapenya · 2020-10-07T15:42:09Z

Before merging this PR, we should make sure that we want exactly this: the artifacts are created without integrity checks, because they will be consumed by internal CI systems

What does this PR do?

It adds the DEV=true env variable when packaging the artifacts, so that they are created in dev mode.

Why is it important?

When using the observability-ci images, which are consumed by different CI projects, the artifacts come with a constraint to check integrity from a .ASC file. Setting this variable will disable that constraint, making it possible to consume the artifacts in a smooth manner on CI time.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

It will create the artifacts with some requirements related to integrity

ph · 2020-10-07T15:51:24Z

.ci/packaging.groovy

+    ]) {
+      dir("${env.BEATS_FOLDER}") {
+        sh(label: "Release ${env.BEATS_FOLDER} ${env.PLATFORMS}", script: 'mage package')
+      }


I am not completely familiar with the CI here, would that run DEV=true mage package for all the beats, quick grep through the code I don't think it would have any impact.

The release() method is called for linux/Mac release: https://github.com/elastic/beats/pull/21645/files#diff-bdaa5a61de14efda6eb2bbb2a38fa9a8R93-R148

DEV doesn't affect mage package AFAIK. Does it affect something else?

You can produce "snapshot" artifacts from packaging with SNAPSHOT=true if that's what you're looking for.

@ph are we ok if we use SNAPSHOT instead?

DEV is used by agent so it skips verification of the artifacts, agent does verify both normal builds and snapshot artifacts. without this artifacts from PRs will fail on these verifications as they wont be signed and 8.0.0-SNAPSHOT artifact published is different
it is a new flag so it should not alter other behavior

Any other concern about @michalpristas's comment? We'd like to merge this to fix some of the errors we see at the e2e tests

ph · 2020-10-07T15:51:53Z

@michalpristas Can you review this too to make sure we don't have side effect?

elasticmachine · 2020-10-07T16:09:06Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #21645 opened]
Start Time: 2020-10-07T15:42:57.695+0000
Duration: 26 min 47 sec

mdelapenya · 2020-10-08T09:51:25Z

@andrewkroh @urso do you see any other implication/concern about this change? We see it super simple to implement, but we also know that the artifacts in the observability-ci namespace are used outside our CI processes.

michalpristas

if this wont affect release builds then it's a ✅ from me

ph · 2020-10-13T17:36:09Z

@mdelapenya LGTM.

mdelapenya · 2020-10-14T09:23:39Z

Merging as per #21645 (comment)

…dependencies-goals * upstream/master: (46 commits) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) typofix for dns timeout configuration option (elastic#21069) chore: create CI artifacts for DEV usage (elastic#21645) [Ingest Manager] Atomic installed forgotten changelog elastic#21780 [Ingest Manager] Agent atomic installer (elastic#21745) Add missing configuration annotations (elastic#21736) [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732) Remove kafka partition ISR from Metricbeat docs (elastic#21709) Skip flaky test with oauth2 config in httpjson input (elastic#21749) Fix for azure retrieve resource by ids (elastic#21711) Update obs app links (elastic#21682) fix: update fleet test suite name (elastic#21738) Remove dot from file.extension value in Auditbeat FIM (elastic#21644) Fix leaks with metadata processors (elastic#16349) Add istiod metricset (elastic#21519) [Ingest Manager] Change Sync/Close call order (elastic#21735) ...

…ne-2.0-arm * upstream/master: (93 commits) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) typofix for dns timeout configuration option (elastic#21069) chore: create CI artifacts for DEV usage (elastic#21645) [Ingest Manager] Atomic installed forgotten changelog elastic#21780 [Ingest Manager] Agent atomic installer (elastic#21745) Add missing configuration annotations (elastic#21736) [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732) Remove kafka partition ISR from Metricbeat docs (elastic#21709) Skip flaky test with oauth2 config in httpjson input (elastic#21749) Fix for azure retrieve resource by ids (elastic#21711) Update obs app links (elastic#21682) fix: update fleet test suite name (elastic#21738) Remove dot from file.extension value in Auditbeat FIM (elastic#21644) Fix leaks with metadata processors (elastic#16349) ...

It will create the artifacts with some requirements related to integrity

* feat: add a new step to run the e2e tests for certain parts of Beats (#21100) * feat: add a new step to run the e2e tests for certain parts of Beats We are going to trigger the tests for those parts affected by the elastic-agent, filebeat, or metricbeat, because those are the ones we verify in the e2e-testing suite * chore: do not include heartbeat * feat: trigger the e2e tests * fix: use relative path * chore: use proper target branch name for PRs * chore: use different tag * fix: use proper env variable * chore: pass github checks context to downstream job * chore: revert shared lib version Co-authored-by: Victor Martinez <[email protected]> * chore: add BASE_DIR env variable Co-authored-by: Victor Martinez <[email protected]> * chore: remove duplicated env * ffix: add param comma separator * fix: wrong copy&paste * chore: move e2e GH check out of the release context * chore: simplify conditional logic * chore: refine execution of test suites * fix: use proper parameter name * chore: set metricbeat version * chore: remove slack notifications on PRs * chore: update parameter * chore: run multiple test suites per beat type Co-authored-by: Victor Martinez <[email protected]> * [E2E Tests] fix: set versions ony for PRs (#21608) * fix: set versions ony for PRs We want to use default versions per branch when running after a merge * fix: add trailing comma Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * [CI: Packaging] fix: push ubi8 images too (#21621) * fix: push ubi8 images too * chore: enhance retries Co-authored-by: Victor Martinez <[email protected]> * chore: use variables in log * chore: add "-oss" images Co-authored-by: Victor Martinez <[email protected]> * fix: remove extra curly brace in script (#21692) * fix: remove extra curly brace * chore: proper indent * fix: update fleet test suite name (#21738) * chore: create CI artifacts for DEV usage (#21645) It will create the artifacts with some requirements related to integrity * chore: simplify triggering the E2E tests for Beats (#21790) * chore: pass beat as a method argument (no side-effects) * chore: run tests in a separate stage * fix: use parenthesis * chore: update comment * chore: do not trigger E2E tests if no suite was added * fix: use missing curly brackets * fix: wrong closure wrapping * fix: condition was not set * chore: delegate variant pushes to the right method (#21861) * fix: delegate pushes to variants * chore: group conditions for x-pack * chore: simplify with endsWith Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * feat: package aliases for snapshots (#21960) * feat: push aliases for docker images * feat: build alias for snapshots * fix: only update alias on snapshots Co-authored-by: Jaime Soriano Pastor <[email protected]> * fix: wrong image name for alias * fix: reuse variable as groovy does not hide variables by scope * chore: extract common logic to a method * Revert "fix: only update alias on snapshots" This reverts commit cff2cef. * Revert "feat: build alias for snapshots" This reverts commit 707e0d7. * chore: do not push aliases for PRs Co-authored-by: Jaime Soriano Pastor <[email protected]> * chore: Use third number as x Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Jaime Soriano Pastor <[email protected]>

* feat: add a new step to run the e2e tests for certain parts of Beats (#21100) * feat: add a new step to run the e2e tests for certain parts of Beats We are going to trigger the tests for those parts affected by the elastic-agent, filebeat, or metricbeat, because those are the ones we verify in the e2e-testing suite * chore: do not include heartbeat * feat: trigger the e2e tests * fix: use relative path * chore: use proper target branch name for PRs * chore: use different tag * fix: use proper env variable * chore: pass github checks context to downstream job * chore: revert shared lib version Co-authored-by: Victor Martinez <[email protected]> * chore: add BASE_DIR env variable Co-authored-by: Victor Martinez <[email protected]> * chore: remove duplicated env * ffix: add param comma separator * fix: wrong copy&paste * chore: move e2e GH check out of the release context * chore: simplify conditional logic * chore: refine execution of test suites * fix: use proper parameter name * chore: set metricbeat version * chore: remove slack notifications on PRs * chore: update parameter * chore: run multiple test suites per beat type Co-authored-by: Victor Martinez <[email protected]> * [E2E Tests] fix: set versions ony for PRs (#21608) * fix: set versions ony for PRs We want to use default versions per branch when running after a merge * fix: add trailing comma Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * [CI: Packaging] fix: push ubi8 images too (#21621) * fix: push ubi8 images too * chore: enhance retries Co-authored-by: Victor Martinez <[email protected]> * chore: use variables in log * chore: add "-oss" images Co-authored-by: Victor Martinez <[email protected]> * fix: remove extra curly brace in script (#21692) * fix: remove extra curly brace * chore: proper indent * fix: update fleet test suite name (#21738) * chore: create CI artifacts for DEV usage (#21645) It will create the artifacts with some requirements related to integrity * chore: simplify triggering the E2E tests for Beats (#21790) * chore: pass beat as a method argument (no side-effects) * chore: run tests in a separate stage * fix: use parenthesis * chore: update comment * chore: do not trigger E2E tests if no suite was added * fix: use missing curly brackets * fix: wrong closure wrapping * fix: condition was not set * chore: delegate variant pushes to the right method (#21861) * fix: delegate pushes to variants * chore: group conditions for x-pack * chore: simplify with endsWith Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> # Conflicts: # .ci/packaging.groovy * feat: package aliases for snapshots (#21960) * feat: push aliases for docker images * feat: build alias for snapshots * fix: only update alias on snapshots Co-authored-by: Jaime Soriano Pastor <[email protected]> * fix: wrong image name for alias * fix: reuse variable as groovy does not hide variables by scope * chore: extract common logic to a method * Revert "fix: only update alias on snapshots" This reverts commit cff2cef. * Revert "feat: build alias for snapshots" This reverts commit 707e0d7. * chore: do not push aliases for PRs Co-authored-by: Jaime Soriano Pastor <[email protected]> * chore: Use third number as x Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Jaime Soriano Pastor <[email protected]>

* feat: add a new step to run the e2e tests for certain parts of Beats (elastic#21100) * feat: add a new step to run the e2e tests for certain parts of Beats We are going to trigger the tests for those parts affected by the elastic-agent, filebeat, or metricbeat, because those are the ones we verify in the e2e-testing suite * chore: do not include heartbeat * feat: trigger the e2e tests * fix: use relative path * chore: use proper target branch name for PRs * chore: use different tag * fix: use proper env variable * chore: pass github checks context to downstream job * chore: revert shared lib version Co-authored-by: Victor Martinez <[email protected]> * chore: add BASE_DIR env variable Co-authored-by: Victor Martinez <[email protected]> * chore: remove duplicated env * ffix: add param comma separator * fix: wrong copy&paste * chore: move e2e GH check out of the release context * chore: simplify conditional logic * chore: refine execution of test suites * fix: use proper parameter name * chore: set metricbeat version * chore: remove slack notifications on PRs * chore: update parameter * chore: run multiple test suites per beat type Co-authored-by: Victor Martinez <[email protected]> * [E2E Tests] fix: set versions ony for PRs (elastic#21608) * fix: set versions ony for PRs We want to use default versions per branch when running after a merge * fix: add trailing comma Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * [CI: Packaging] fix: push ubi8 images too (elastic#21621) * fix: push ubi8 images too * chore: enhance retries Co-authored-by: Victor Martinez <[email protected]> * chore: use variables in log * chore: add "-oss" images Co-authored-by: Victor Martinez <[email protected]> * fix: remove extra curly brace in script (elastic#21692) * fix: remove extra curly brace * chore: proper indent * fix: update fleet test suite name (elastic#21738) * chore: create CI artifacts for DEV usage (elastic#21645) It will create the artifacts with some requirements related to integrity * chore: simplify triggering the E2E tests for Beats (elastic#21790) * chore: pass beat as a method argument (no side-effects) * chore: run tests in a separate stage * fix: use parenthesis * chore: update comment * chore: do not trigger E2E tests if no suite was added * fix: use missing curly brackets * fix: wrong closure wrapping * fix: condition was not set * chore: delegate variant pushes to the right method (elastic#21861) * fix: delegate pushes to variants * chore: group conditions for x-pack * chore: simplify with endsWith Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * feat: package aliases for snapshots (elastic#21960) * feat: push aliases for docker images * feat: build alias for snapshots * fix: only update alias on snapshots Co-authored-by: Jaime Soriano Pastor <[email protected]> * fix: wrong image name for alias * fix: reuse variable as groovy does not hide variables by scope * chore: extract common logic to a method * Revert "fix: only update alias on snapshots" This reverts commit cff2cef. * Revert "feat: build alias for snapshots" This reverts commit 707e0d7. * chore: do not push aliases for PRs Co-authored-by: Jaime Soriano Pastor <[email protected]> * chore: Use third number as x Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Jaime Soriano Pastor <[email protected]> (cherry picked from commit 624c459)

…23142) Backports the following commits to 7.x: * feat: add a new step to run the e2e tests for certain parts of Beats (#21100) * [E2E Tests] fix: set versions ony for PRs (#21608) * [CI: Packaging] fix: push ubi8 images too (#21621) * fix: remove extra curly brace in script (#21692) * fix: update fleet test suite name (#21738) * chore: create CI artifacts for DEV usage (#21645) * chore: simplify triggering the E2E tests for Beats (#21790) * chore: delegate variant pushes to the right method (#21861) * feat: package aliases for snapshots (#21960) * fix: use proper param name for e2e tests (#22836) Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]>

…ports (elastic#23142) Backports the following commits to 7.x: * feat: add a new step to run the e2e tests for certain parts of Beats (elastic#21100) * [E2E Tests] fix: set versions ony for PRs (elastic#21608) * [CI: Packaging] fix: push ubi8 images too (elastic#21621) * fix: remove extra curly brace in script (elastic#21692) * fix: update fleet test suite name (elastic#21738) * chore: create CI artifacts for DEV usage (elastic#21645) * chore: simplify triggering the E2E tests for Beats (elastic#21790) * chore: delegate variant pushes to the right method (elastic#21861) * feat: package aliases for snapshots (elastic#21960) * fix: use proper param name for e2e tests (elastic#22836) Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 1006bd9)

…23142) (#23396) Backports the following commits to 7.x: * feat: add a new step to run the e2e tests for certain parts of Beats (#21100) * [E2E Tests] fix: set versions ony for PRs (#21608) * [CI: Packaging] fix: push ubi8 images too (#21621) * fix: remove extra curly brace in script (#21692) * fix: update fleet test suite name (#21738) * chore: create CI artifacts for DEV usage (#21645) * chore: simplify triggering the E2E tests for Beats (#21790) * chore: delegate variant pushes to the right method (#21861) * feat: package aliases for snapshots (#21960) * fix: use proper param name for e2e tests (#22836) (cherry picked from commit 1006bd9) Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]>

* feat: add a new step to run the e2e tests for certain parts of Beats (elastic#21100) * feat: add a new step to run the e2e tests for certain parts of Beats We are going to trigger the tests for those parts affected by the elastic-agent, filebeat, or metricbeat, because those are the ones we verify in the e2e-testing suite * chore: do not include heartbeat * feat: trigger the e2e tests * fix: use relative path * chore: use proper target branch name for PRs * chore: use different tag * fix: use proper env variable * chore: pass github checks context to downstream job * chore: revert shared lib version Co-authored-by: Victor Martinez <[email protected]> * chore: add BASE_DIR env variable Co-authored-by: Victor Martinez <[email protected]> * chore: remove duplicated env * ffix: add param comma separator * fix: wrong copy&paste * chore: move e2e GH check out of the release context * chore: simplify conditional logic * chore: refine execution of test suites * fix: use proper parameter name * chore: set metricbeat version * chore: remove slack notifications on PRs * chore: update parameter * chore: run multiple test suites per beat type Co-authored-by: Victor Martinez <[email protected]> * [E2E Tests] fix: set versions ony for PRs (elastic#21608) * fix: set versions ony for PRs We want to use default versions per branch when running after a merge * fix: add trailing comma Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> * [CI: Packaging] fix: push ubi8 images too (elastic#21621) * fix: push ubi8 images too * chore: enhance retries Co-authored-by: Victor Martinez <[email protected]> * chore: use variables in log * chore: add "-oss" images Co-authored-by: Victor Martinez <[email protected]> * fix: remove extra curly brace in script (elastic#21692) * fix: remove extra curly brace * chore: proper indent * fix: update fleet test suite name (elastic#21738) * chore: create CI artifacts for DEV usage (elastic#21645) It will create the artifacts with some requirements related to integrity * chore: simplify triggering the E2E tests for Beats (elastic#21790) * chore: pass beat as a method argument (no side-effects) * chore: run tests in a separate stage * fix: use parenthesis * chore: update comment * chore: do not trigger E2E tests if no suite was added * fix: use missing curly brackets * fix: wrong closure wrapping * fix: condition was not set * chore: delegate variant pushes to the right method (elastic#21861) * fix: delegate pushes to variants * chore: group conditions for x-pack * chore: simplify with endsWith Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Victor Martinez <[email protected]> # Conflicts: # .ci/packaging.groovy * feat: package aliases for snapshots (elastic#21960) * feat: push aliases for docker images * feat: build alias for snapshots * fix: only update alias on snapshots Co-authored-by: Jaime Soriano Pastor <[email protected]> * fix: wrong image name for alias * fix: reuse variable as groovy does not hide variables by scope * chore: extract common logic to a method * Revert "fix: only update alias on snapshots" This reverts commit cff2cef. * Revert "feat: build alias for snapshots" This reverts commit 707e0d7. * chore: do not push aliases for PRs Co-authored-by: Jaime Soriano Pastor <[email protected]> * chore: Use third number as x Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Jaime Soriano Pastor <[email protected]>

chore: create CI artifacts for DEV usage

c79a37e

It will create the artifacts with some requirements related to integrity

mdelapenya self-assigned this Oct 7, 2020

mdelapenya requested a review from a team October 7, 2020 15:42

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 7, 2020

mdelapenya requested a review from ph October 7, 2020 15:42

mdelapenya added the Team:Automation Label for the Observability productivity team label Oct 7, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 7, 2020

ph reviewed Oct 7, 2020

View reviewed changes

ph requested review from michalpristas and ph October 7, 2020 15:51

andresrc added the [zube]: Inbox label Oct 7, 2020

kuisathaverat approved these changes Oct 8, 2020

View reviewed changes

mdelapenya marked this pull request as ready for review October 8, 2020 08:42

michalpristas approved these changes Oct 13, 2020

View reviewed changes

ph approved these changes Oct 13, 2020

View reviewed changes

cachedout approved these changes Oct 14, 2020

View reviewed changes

mdelapenya merged commit e3cf993 into elastic:master Oct 14, 2020

zube bot added [zube]: Done and removed [zube]: Inbox labels Oct 14, 2020

mdelapenya mentioned this pull request Oct 22, 2020

Cherry-pick to 7.10: packaging backports #22076

Merged

mdelapenya added a commit to mdelapenya/beats that referenced this pull request Oct 22, 2020

chore: create CI artifacts for DEV usage (elastic#21645)

5f1e944

It will create the artifacts with some requirements related to integrity

mdelapenya mentioned this pull request Oct 22, 2020

Cherry-pick to 7.9: packaging backports #22078

Merged

mdelapenya added a commit to mdelapenya/beats that referenced this pull request Oct 22, 2020

chore: create CI artifacts for DEV usage (elastic#21645)

d00d3d1

It will create the artifacts with some requirements related to integrity

jsoriano mentioned this pull request Dec 15, 2020

Cherry-pick #22076 to 7.x: Cherry-pick to 7.10: packaging backports #23142

Merged

jsoriano mentioned this pull request Jan 7, 2021

Cherry-pick #23142 to 7.11: Cherry-pick #22076 to 7.x: Cherry-pick to 7.10: packaging backports #23396

Merged

zube bot removed the [zube]: Done label Jan 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: create CI artifacts for DEV usage #21645

chore: create CI artifacts for DEV usage #21645

mdelapenya commented Oct 7, 2020 •

edited by zube bot

Loading

ph Oct 7, 2020

mdelapenya Oct 7, 2020

andrewkroh Oct 8, 2020

mdelapenya Oct 8, 2020

michalpristas Oct 13, 2020 •

edited

Loading

mdelapenya Oct 13, 2020

ph commented Oct 7, 2020

elasticmachine commented Oct 7, 2020

Build stats

mdelapenya commented Oct 8, 2020

michalpristas left a comment •

edited

Loading

ph commented Oct 13, 2020

mdelapenya commented Oct 14, 2020

chore: create CI artifacts for DEV usage #21645

chore: create CI artifacts for DEV usage #21645

Conversation

mdelapenya commented Oct 7, 2020 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

ph Oct 7, 2020

Choose a reason for hiding this comment

mdelapenya Oct 7, 2020

Choose a reason for hiding this comment

andrewkroh Oct 8, 2020

Choose a reason for hiding this comment

mdelapenya Oct 8, 2020

Choose a reason for hiding this comment

michalpristas Oct 13, 2020 • edited Loading

Choose a reason for hiding this comment

mdelapenya Oct 13, 2020

Choose a reason for hiding this comment

ph commented Oct 7, 2020

elasticmachine commented Oct 7, 2020

💚 Build Succeeded

Build stats

mdelapenya commented Oct 8, 2020

michalpristas left a comment • edited Loading

Choose a reason for hiding this comment

ph commented Oct 13, 2020

mdelapenya commented Oct 14, 2020

mdelapenya commented Oct 7, 2020 •

edited by zube bot

Loading

michalpristas Oct 13, 2020 •

edited

Loading

michalpristas left a comment •

edited

Loading