Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Pipeline 2.0 for monorepos #20104

Merged
merged 112 commits into from
Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
ac465eb
[CI] Pipeline 2.0 for monorepos
v1v Jul 21, 2020
aa6a07a
Update auditbeat/Jenkinsfile.yml
v1v Jul 23, 2020
1b91c12
Update auditbeat/Jenkinsfile.yml
v1v Jul 23, 2020
8175b5a
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Jul 27, 2020
7183f93
[CI] 2.0 YAML to dynamic stage generations
v1v Jul 27, 2020
4290714
Merge branch 'feature/ci-pipeline-2.0' of github.com:v1v/beats into f…
v1v Jul 27, 2020
ca9a5a6
Use shared library
v1v Jul 27, 2020
56c45d5
Use beatsWhen and beatsStages
v1v Jul 28, 2020
778da9e
Revert "Use beatsWhen and beatsStages"
v1v Jul 28, 2020
daf8fb1
Simplify
v1v Jul 28, 2020
fcab64b
Refactor
v1v Jul 28, 2020
6a0f1bd
Add run commands
v1v Jul 28, 2020
a8cc866
use nodeOS step
v1v Jul 28, 2020
14e039d
Fix GO_VERSION
v1v Jul 28, 2020
7660a6d
Transform to String
v1v Jul 28, 2020
35d30be
Use method-pointer-operator
v1v Jul 28, 2020
6be81e2
replace with isInstalled
v1v Jul 28, 2020
e874d54
Fix the YAML data
v1v Jul 28, 2020
05baf2e
Use node
v1v Jul 28, 2020
2a668df
sort variables
v1v Jul 28, 2020
b22f34a
Use string rather than a list
v1v Jul 29, 2020
8126103
[CI] verbose and added some missing functions
v1v Jul 29, 2020
c5a55db
Use the right arguments
v1v Jul 29, 2020
02e0303
Use the directory flag in the make
v1v Jul 29, 2020
11653c0
Revert "Use the directory flag in the make"
v1v Jul 29, 2020
52ec93e
directoryFlag not required
v1v Jul 29, 2020
1412871
Archive build reasons
v1v Jul 29, 2020
bd853af
Support windows builds
v1v Jul 29, 2020
bc5a6a5
Tear down environment context in the workers
v1v Jul 29, 2020
2df49f8
[CI] support x-pack/auditbeat and minor changes
v1v Jul 29, 2020
40723af
Support MODULE env variable
v1v Jul 29, 2020
a3d9a5c
Build reasons pattern file changed
v1v Jul 29, 2020
8902166
Support runbld and test reporting
v1v Jul 29, 2020
21b7876
[markdown] add PR comments that are supported
v1v Jul 29, 2020
e6acb6f
[CI] Update README.md with CI Labels
v1v Jul 29, 2020
9f2b923
Fix step reference
v1v Jul 29, 2020
09262e5
Disable tests for the x-pack/auditbeat
v1v Jul 29, 2020
6e1e068
Support filebeat
v1v Jul 29, 2020
08d0d4a
Support x-pack/filebeat
v1v Jul 29, 2020
4a6f8d1
Support packetbeat
v1v Jul 29, 2020
35dfdb4
Support metricbeat
v1v Jul 29, 2020
f00ca46
[IGNORE] unitTest for the timebeing
v1v Jul 29, 2020
c6c9ea8
Add configuration for the remaining beats
v1v Jul 30, 2020
3ebe3e0
Refactor getCommonModuleInTheChangeSet
v1v Jul 30, 2020
aa672c3
[CI] Cosmetic change
v1v Jul 30, 2020
5867081
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Jul 30, 2020
abd6589
Fix leftover and add comments
v1v Jul 30, 2020
1d4ea16
Fix leftovers
v1v Jul 30, 2020
1e3d9b2
Disable linting for the time being with some TODO
v1v Jul 30, 2020
4e1d3a1
[CI] Script to generate the build-stage table
v1v Jul 30, 2020
4b8466b
[docs] Add when in the table markdown
v1v Jul 30, 2020
a361e8d
[CI] support deploy/kubernetes
v1v Jul 30, 2020
ce7e02b
[CI] support cloud aws for x-pack/metricbeats
v1v Jul 30, 2020
f2cdc17
Fix typo
v1v Jul 30, 2020
38ea5dd
[CI] It uses another node
v1v Jul 30, 2020
f3beedc
Use context that's populated from the beatsStages step
v1v Jul 30, 2020
f99006c
[DOCS] Add when column with some legend
v1v Jul 30, 2020
eff00d7
No yet supported
v1v Jul 30, 2020
d0d8d73
Enable AWS testing for the time being
v1v Jul 30, 2020
6e381fa
For debugging purposes
v1v Jul 30, 2020
d80a490
Dump and support x-pack/winlogbeat
v1v Jul 30, 2020
f9bd227
Refactor makeTarget and mageTarget with target
v1v Jul 30, 2020
ac360da
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Jul 30, 2020
3936430
Fix typo in the variable name
v1v Jul 30, 2020
ae322e3
Support getProjectDependencies in the beatsWhen
v1v Jul 30, 2020
7b8822c
no debug in the CI for now
v1v Jul 30, 2020
83efbc5
Terraform leftovers in the CI
v1v Jul 30, 2020
76a90ae
Add missing goals since tests are not failing anymore
v1v Jul 31, 2020
1624b54
Support testGCPFunctions
v1v Jul 31, 2020
13fb5f0
Set the project dependencies for the generators using the special tok…
v1v Jul 31, 2020
5a22d2f
Use eval gvm to avoid hardcoded PATHs
v1v Jul 31, 2020
e25ba73
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Jul 31, 2020
b5f99de
Fix linting issues with autopep8
v1v Jul 31, 2020
269dee5
Folder is not required since the caller already configured it
v1v Jul 31, 2020
196e6bd
Fix runbld issue
v1v Jul 31, 2020
5785585
Refactor a bit the cloud target
v1v Jul 31, 2020
4e27aa1
Refactor getProjectDependencies
v1v Jul 31, 2020
414a8f0
Use ubuntu-18 and disable windows-2016
v1v Jul 31, 2020
9e97630
[runbld] debug what's going on
v1v Jul 31, 2020
f4cd74c
Avoid to error if no Go files in
v1v Jul 31, 2020
7339f12
[ci] use previous approach to support docker for mac issues
v1v Jul 31, 2020
9727b52
[POC] Prepare context for the skip/draft PRs
v1v Jul 31, 2020
a492e6b
[RUNBLD] force JOB-NAME to match
v1v Jul 31, 2020
35b6b37
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Jul 31, 2020
6fa6d3a
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Aug 6, 2020
5893c8f
[CI] runbld support
v1v Aug 6, 2020
f701834
[CI] force cwd and pass the id args
v1v Aug 6, 2020
0395eb8
fixes The most common cause is that Jenkins and runbld are configured…
v1v Aug 6, 2020
2f21086
Tests need to be stored in the workspace
v1v Aug 6, 2020
23c69ce
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Aug 12, 2020
d431b02
Test AWS
v1v Aug 12, 2020
1faa11d
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 2, 2020
993f336
Revert "Test AWS"
v1v Sep 3, 2020
46c3ad9
[CI] Fix params name
v1v Sep 3, 2020
9ce54de
Disable cloud as long as https://github.com/elastic/beats/issues/2095…
v1v Sep 3, 2020
a3da232
Fix wound up catching method mismatch with this.& by using a Class-
v1v Sep 3, 2020
289d5ff
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 3, 2020
df6ebd5
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 3, 2020
b08c40a
Enable cloud stages
v1v Sep 3, 2020
bba848d
Apply suggestions from code review
v1v Sep 14, 2020
bb7cd78
Update .ci/Jenkinsfile
v1v Sep 14, 2020
61621c5
Update .ci/Jenkinsfile
v1v Sep 14, 2020
34955f2
As commented in the review
v1v Sep 15, 2020
987d880
As commented in the review
v1v Sep 15, 2020
8374f56
Enable target without a node
v1v Sep 16, 2020
5bda33f
Revert "Enable target without a node"
v1v Sep 17, 2020
b4a1faf
Simplify the lint stage compared to the previous approach
v1v Sep 17, 2020
4c7e8da
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 17, 2020
20788d9
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 18, 2020
32f0c6c
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 18, 2020
d636f2c
Replace Jenkinsfile with .ci/Jenkinsfile. Remove jjbb
v1v Sep 24, 2020
778c037
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
v1v Sep 24, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
202 changes: 202 additions & 0 deletions .ci/Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
#!/usr/bin/env groovy

@Library('apm@current') _

pipeline {
agent { label 'ubuntu-18 && immutable' }
environment {
AWS_ACCOUNT_SECRET = 'secret/observability-team/ci/elastic-observability-aws-account-auth'
BASE_DIR = 'src/github.com/elastic/beats'
DOCKERELASTIC_SECRET = 'secret/observability-team/ci/docker-registry/prod'
DOCKER_COMPOSE_VERSION = "1.21.0"
DOCKER_REGISTRY = 'docker.elastic.co'
GOX_FLAGS = "-arch amd64"
JOB_GCS_BUCKET = 'beats-ci-temp'
JOB_GCS_CREDENTIALS = 'beats-ci-gcs-plugin'
PIPELINE_LOG_LEVEL = 'INFO'
OSS_MODULE_PATTERN = '^[a-z0-9]+beat\\/module\\/([^\\/]+)\\/.*'
RUNBLD_DISABLE_NOTIFICATIONS = 'true'
TERRAFORM_VERSION = "0.12.24"
XPACK_MODULE_PATTERN = '^x-pack\\/[a-z0-9]+beat\\/module\\/([^\\/]+)\\/.*'
}
options {
timeout(time: 2, unit: 'HOURS')
buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '20', daysToKeepStr: '30'))
timestamps()
ansiColor('xterm')
disableResume()
durabilityHint('PERFORMANCE_OPTIMIZED')
quietPeriod(10)
rateLimitBuilds(throttle: [count: 60, durationName: 'hour', userBoost: true])
}
triggers {
issueCommentTrigger('(?i)(.*(?:jenkins\\W+)?run\\W+(?:the\\W+)?tests(?:\\W+please)?.*|^/test\\W+.*$)')
}
parameters {
booleanParam(name: 'allCloudTests', defaultValue: false, description: 'Run all cloud integration tests.')
booleanParam(name: 'awsCloudTests', defaultValue: false, description: 'Run AWS cloud integration tests.')
string(name: 'awsRegion', defaultValue: 'eu-central-1', description: 'Default AWS region to use for testing.')
booleanParam(name: 'runAllStages', defaultValue: false, description: 'Allow to run all stages.')
booleanParam(name: 'macosTest', defaultValue: false, description: 'Allow macOS stages.')
booleanParam(name: 'windowsTest', defaultValue: true, description: 'Allow Windows stages.')
booleanParam(name: 'debug', defaultValue: false, description: 'Allow debug logging for Jenkins steps')
}
stages {
stage('Checkout') {
options { skipDefaultCheckout() }
steps {
pipelineManager([ cancelPreviousRunningBuilds: [ when: 'PR' ] ])
deleteDir()
gitCheckout(basedir: "${BASE_DIR}", githubNotifyFirstTimeContributor: true)
stashV2(name: 'source', bucket: "${JOB_GCS_BUCKET}", credentialsId: "${JOB_GCS_CREDENTIALS}")
script {
dir("${BASE_DIR}"){
// Skip all the stages except docs for PR's with asciidoc and md changes only
env.ONLY_DOCS = isGitRegionMatch(patterns: [ '.*\\.(asciidoc|md)' ], shouldMatchAll: true)
}
}
}
}
stage('Lint'){
options { skipDefaultCheckout() }
environment {
GOFLAGS = '-mod=readonly'
}
when {
anyOf {
not { changeRequest() } // If no PR
allOf { // If PR and no docs changes
expression { return env.ONLY_DOCS == "false" }
jsoriano marked this conversation as resolved.
Show resolved Hide resolved
changeRequest()
}
expression { return params.runAllStages } // If UI forced
}
}
steps {
echo 'linting'
}
}
stage('Build&Test') {
options { skipDefaultCheckout() }
steps {
deleteDir()
unstash "source"
dir("${BASE_DIR}"){
script {
def masterFile = 'Jenkinsfile.yml'
def changeset = readYaml(file: masterFile)['changeset']
readYaml(file: masterFile)['projects'].each { project ->
generateSteps(name: project, changeset: changeset).each { k,v ->
mapParallelTasks["${k}"] = v
}
}
parallel(mapParallelTasks)
}
}
}
}
}
post {
// TODO: Support runbld
cleanup {
notifyBuildResult(prComment: true)
}
}
}

def generateSteps(Map args = [:]) {
def name = args.project
def changeset = args.changeset
def mapParallelStages = [:]
def masterFile = "${name}/Jenkinsfile.yml"
def content = readYaml(file: masterFile)
def defaultNode = content['platform']

if (whenValidation(project: name, content: content, changeset: changeset)) {
content['stages'].each { stageName ->
if (stageName['when']) {
if (whenValidation(project: "stageName", content: stageName['when'], changeset: changeset)) {
generateStages(defaultNode: defaultNode, content: stageName).each { k,v ->
mapParallelStages["${k}"] = v
}
}
} else {
generateStages(defaultNode: defaultNode, content: stageName).each { k,v ->
mapParallelStages["${k}"] = v
}
}
}
return mapParallelStages
}

def generateStages(Map args = [:]) {
def defaultNode = args.defaultNode
def mapParallelStages = [:]
if (content['platform']) {
content['platform']?.each {
mapParallelStages["${k}-${it}"] = generateStage(platform: it, content: stageName)
}
} else {
mapParallelStages["${k}"] = generateStage(platform: defaultNode, content: stageName)
}
return mapParallelStages
}

def generateStage(Map args = [:]) {
return {
node(it) {
sh "${content['command']}"
v1v marked this conversation as resolved.
Show resolved Hide resolved
}
}
}

/**
* Given the YAML definition and the changeset global macros
* then it verifies if the project or stage should be enabled.
*
* It will generate a
*/
def whenValidation(Map args = [:]) {
def project = args.project
def content = args.content
def patterns = args.changeset
def ret = false
if (content['comments']?.trim() && env.GITHUB_COMMENT?.trim()) {
if (content['comments']?.any { env.GITHUB_COMMENT?.toLowerCase()?.contains(it?.toLowerCase()) }) {
ret = true
markdownReason(project: project, reason: 'Comment is enabled and matches with the pattern.')
} else {
markdownReason(project: project, reason: 'Comment is enabled and does not match with the pattern.')
}
}
if (content['labels']) {
if (content['labels']?.any { matchesPrLabel(label: it) }) {
ret = true
markdownReason(project: project, reason: 'Label is enabled and matches with the pattern.')
} else {
markdownReason(project: project, reason: 'Label is enabled and does not match with the pattern.')
}
}
if (content['parameters']) {
if (content['parameters']?.any { params.[it] }) {
ret = true
markdownReason(project: project, reason: 'Parameter is enabled and matches with the pattern.')
} else {
markdownReason(project: project, reason: 'Parameter is enabled and does not match with the pattern.')
}
}
if (content['branches'] && env.BRANCH_NAME?.trim()) {
ret = true
markdownReason(project: project, reason: 'Branch is enabled and matches with the pattern.')
}
if (content['tags'] && env.TAG_NAME?.trim()) {
ret = true
markdownReason(project: project, reason: 'Tag is enabled and matches with the pattern.')
}
return ret
}

def markdownReason(Map args = [:]) {
echo "${args.project} - ${args.reason}"
// TODO create markdown
}
60 changes: 60 additions & 0 deletions .ci/jobs/beats-mbp-2.0.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
- job:
name: Beats/beats-mbp-2.0
display-name: 'Beats (2.0)'
description: 'Beats Main Pipeline 2.0'
view: Beats
concurrent: true
project-type: multibranch
prune-dead-branches: true
days-to-keep: 30
script-path: '.ci/Jenkinsfile'
triggers: []
wrappers: []
scm:
- github:
branch-discovery: 'no-pr'
discover-pr-forks-strategy: 'merge-current'
discover-pr-forks-trust: 'permission'
discover-pr-origin: 'merge-current'
head-filter-regex: '(master|7\.[x789]|8\.\d+|PR-.*|v\d+\.\d+\.\d+)'
discover-tags: true
disable-pr-notifications: true
notification-context: "beats-ci-2.0"
repo: 'beats'
repo-owner: 'elastic'
credentials-id: github-app-beats-ci
ssh-checkout:
credentials: f6c7695a-671e-4f4f-a331-acdce44ff9ba
build-strategies:
- skip-initial-build: true
- tags:
ignore-tags-older-than: -1
ignore-tags-newer-than: 365
- change-request:
ignore-target-only-changes: true
- named-branches:
- exact-name:
name: 'master'
case-sensitive: true
- regex-name:
regex: '7\.[x789]'
case-sensitive: true
- regex-name:
regex: '8\.\d+'
case-sensitive: true
clean:
after: true
before: true
prune: true
shallow-clone: true
depth: 3
do-not-fetch-tags: true
submodule:
disable: false
recursive: true
parent-credentials: true
timeout: 100
timeout: '15'
use-author: true
wipe-workspace: true
29 changes: 29 additions & 0 deletions Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
projects:
- "auditbeat"
- "filebeat"
- "heartbeat"
- "journalbeat"
- "libbeat"
- "metricbeat"
- "packetbeat"
- "winlogbeat"
- "x-pack/auditbeat"
- "x-pack/dockerlogbeat"
- "x-pack/filebeat"
- "x-pack/functionbeat"
- "x-pack/heartbeat"
- "x-pack/journalbeat"
- "x-pack/libbeat"
- "x-pack/metricbeat"
- "x-pack/packetbeat"
- "x-pack/winlogbeat"

## Changeset macros that are defined here and used in each specific 2.0 pipeline.
changeset:
oss:
- "^Jenkinsfile"
- "^go.mod"
- "^libbeat/.*"
- "^testing/.*"
- "^dev-tools/.*"
- "^\\.ci/scripts/.*"
44 changes: 44 additions & 0 deletions auditbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
when:
branches: true ## for all the branches
tags: true ## for all the tags
changeset: ## when PR contains any of those entries in the changeset
- "auditbeat"
- '@oss' ## special token regarding the changeset for the oss
comments:
- "/test auditbeat"
labels:
- "auditbeat"
parameters:
- "auditbeat"
platform: ## default platform where the stages will use
- "linux && ubuntu-16"
stages:
build:
command:
- "cd auditbeat && mage build test"
crosscompile:
command:
- "make -C auditbeat crosscompile"
macos:
command:
- "cd auditbeat && mage build unitTest"
v1v marked this conversation as resolved.
Show resolved Hide resolved
platform:
- "macos"
when: ## Aggregate when with the top-level one.
comments:
- "/test auditbeat for macos"
labels:
- "macos"
parameters:
- "macos"
windows:
command:
- "cd auditbeat && mage build unitTest"
platform:
- "windows-2019"
- "windows-2016"
when:
comments:
- "/test auditbeat for windows"
parameters:
- "macos"