Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests" #26704

Merged
merged 1 commit into from
Jul 5, 2021
Merged

Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests" #26704

merged 1 commit into from
Jul 5, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Jul 5, 2021 •
edited
Loading

Reverts #26617

Why

Unfortunately, there are a couple of issues:

  1. Test population with the retry does not reset the test failures.
  2. Dynamic stage does not reset the overall build status.

Somehow the tests I ran with the original PR always worked, so I could not detect this behaviour.

Alternative for this approach:

@v1v v1v requested a review from a team as a code owner July 5, 2021 08:54
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 5, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 5, 2021 •
edited by jenkins-beats-ci bot
Loading

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: v1v commented: /test

  • Start Time: 2021-07-05T11:46:06.002+0000

  • Duration: 43 min 15 sec

  • Commit: d3cdebc

Test stats 🧪

Test Results
Failed 0
Passed 29379
Skipped 3892
Total 33271

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

Google Storage Download
  • Took 0 min 20 sec . View more details on here
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details on here

Log output

Expand to view the last 100 lines of log output 

[2021-07-05T12:23:43.387Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwZDNjZGViYzBiYzc5NDU4YmQ4YWM3OTdjZjExODZjMDk3ZGI0ZTE3Ng gs://beats-ci-temp/ci/cache/
[2021-07-05T12:23:43.481Z] + gsutil --version
[2021-07-05T12:23:44.892Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-07-05T12:23:45.202Z] + gcloud auth activate-service-account --key-file ****
[2021-07-05T12:23:45.770Z] Activated service account credentials for: [[email protected]]
[2021-07-05T12:23:46.081Z] + gsutil -m -q cp -a public-read eC1wYWNrL21ldHJpY2JlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwZDNjZGViYzBiYzc5NDU4YmQ4YWM3OTdjZjExODZjMDk3ZGI0ZTE3Ng gs://beats-ci-temp/ci/cache/
[2021-07-05T12:24:41.861Z] tests\system\test_processors.py ........                                 [ 13%]
[2021-07-05T12:25:14.138Z] tests\system\test_reload.py ...                                          [ 14%]
[2021-07-05T12:25:14.138Z] tests\system\test_template.py s                                          [ 15%]
[2021-07-05T12:25:22.286Z] tests\system\test_timeseries.py .                                        [ 15%]
[2021-07-05T12:25:22.286Z] module\aerospike\test_aerospike.py s                                     [ 16%]
[2021-07-05T12:25:22.287Z] module\apache\test_apache.py ss                                          [ 17%]
[2021-07-05T12:25:22.287Z] module\ceph\test_ceph.py ssssssssssssssssssss                            [ 25%]
[2021-07-05T12:25:22.287Z] module\consul\test_consul.py ss                                          [ 25%]
[2021-07-05T12:25:22.287Z] module\couchbase\test_couchbase.py sss                                   [ 27%]
[2021-07-05T12:25:22.287Z] module\couchdb\test_couchdb.py s                                         [ 27%]
[2021-07-05T12:25:22.287Z] module\docker\test_docker.py sssssssss                                   [ 31%]
[2021-07-05T12:25:22.287Z] module\dropwizard\test_dropwizard.py sss                                 [ 32%]
[2021-07-05T12:25:22.287Z] module\elasticsearch\test_elasticsearch.py sssssssssss                   [ 36%]
[2021-07-05T12:25:22.287Z] module\envoyproxy\test_envoyproxy.py ss                                  [ 37%]
[2021-07-05T12:25:22.287Z] module\etcd\test_etcd.py ssssssss                                        [ 40%]
[2021-07-05T12:25:22.287Z] module\golang\test_golang.py s                                           [ 41%]
[2021-07-05T12:25:22.287Z] module\haproxy\test_haproxy.py sssssssssssssssssssssssss                 [ 51%]
[2021-07-05T12:25:32.290Z] module\http\test_http.py s.                                              [ 52%]
[2021-07-05T12:25:32.290Z] module\jolokia\test_jolokia.py ssss                                      [ 53%]
[2021-07-05T12:25:32.290Z] module\kafka\test_kafka.py ssssssssssssssssssss                          [ 61%]
[2021-07-05T12:25:32.290Z] module\kibana\test_kibana.py ss                                          [ 62%]
[2021-07-05T12:25:32.290Z] module\logstash\test_logstash.py sss                                     [ 63%]
[2021-07-05T12:25:32.290Z] module\memcached\test_memcached.py s                                     [ 64%]
[2021-07-05T12:25:32.290Z] module\mongodb\test_mongodb.py s                                         [ 64%]
[2021-07-05T12:25:32.290Z] module\munin\test_munin.py s                                             [ 65%]
[2021-07-05T12:25:32.290Z] module\mysql\test_mysql.py sssssss                                       [ 68%]
[2021-07-05T12:25:32.290Z] module\nats\test_nats.py ssssssssssssssssss                              [ 75%]
[2021-07-05T12:25:32.290Z] module\openmetrics\test_openmetrics.py s                                 [ 75%]
[2021-07-05T12:25:32.290Z] module\php_fpm\test_phpfpm.py s                                          [ 76%]
[2021-07-05T12:25:32.290Z] module\postgresql\test_postgresql.py ssssssssssssssssssss                [ 84%]
[2021-07-05T12:25:32.290Z] module\prometheus\test_prometheus.py sss                                 [ 85%]
[2021-07-05T12:25:32.290Z] module\redis\test_redis.py ssssssssssss                                  [ 90%]
[2021-07-05T12:27:29.772Z] module\system\test_system.py .....s..s...s.s.                            [ 96%]
[2021-07-05T12:27:29.772Z] module\traefik\test_traefik.py s                                         [ 97%]
[2021-07-05T12:27:29.772Z] module\uwsgi\test_uwsgi.py ss                                            [ 97%]
[2021-07-05T12:27:29.772Z] module\vsphere\test_vsphere.py sss                                       [ 99%]
[2021-07-05T12:27:29.772Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-07-05T12:27:29.772Z] 
[2021-07-05T12:27:29.772Z] ============================== warnings summary ===============================
[2021-07-05T12:27:29.772Z] tests\system\test_lightmodules.py:57
[2021-07-05T12:27:29.772Z]   C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-07-05T12:27:29.772Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-07-05T12:27:29.772Z] 
[2021-07-05T12:27:29.772Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-07-05T12:27:29.772Z] - generated xml file: C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-07-05T12:27:29.772Z] ============================ slowest 20 durations =============================
[2021-07-05T12:27:29.773Z] 17.02s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-07-05T12:27:29.773Z] 17.00s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-07-05T12:27:29.773Z] 16.97s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-07-05T12:27:29.773Z] 14.04s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-07-05T12:27:29.773Z] 12.67s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-07-05T12:27:29.773Z] 11.65s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-07-05T12:27:29.773Z] 10.97s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-07-05T12:27:29.773Z] 10.88s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-07-05T12:27:29.773Z] 9.64s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-07-05T12:27:29.773Z] 9.24s call     metricbeat/tests/system/test_processors.py::Test::test_multiple_actions
[2021-07-05T12:27:29.773Z] 9.21s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_complex_condition
[2021-07-05T12:27:29.773Z] 9.19s call     metricbeat/tests/system/test_processors.py::Test::test_dropfields_with_condition
[2021-07-05T12:27:29.773Z] 9.18s call     metricbeat/tests/system/test_processors.py::Test::test_include_fields
[2021-07-05T12:27:29.773Z] 9.15s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-07-05T12:27:29.773Z] 9.13s call     metricbeat/tests/system/test_processors.py::Test::test_contradictory_multiple_actions
[2021-07-05T12:27:29.773Z] 9.08s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-07-05T12:27:29.773Z] 9.07s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-07-05T12:27:29.773Z] 8.98s call     metricbeat/tests/system/test_base.py::Test::test_export_config
[2021-07-05T12:27:29.773Z] 8.94s call     metricbeat/module/http/test_http.py::Test::test_server
[2021-07-05T12:27:29.773Z] 8.92s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-07-05T12:27:29.773Z] =========== 40 passed, 207 skipped, 1 warning in 407.38s (0:06:47) ============
[2021-07-05T12:27:29.773Z] >> python test: Unit Testing Complete
[2021-07-05T12:27:30.130Z] 
[2021-07-05T12:27:30.130Z] C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-07-05T12:27:33.002Z] 
[2021-07-05T12:27:33.002Z] C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-07-05T12:27:34.927Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-07-05T12:27:34.927Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-07-05T12:27:34.945Z] Running in C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats\build
[2021-07-05T12:27:34.963Z] Recording test results
[2021-07-05T12:27:37.274Z] [Checks API] No suitable checks publisher found.
[2021-07-05T12:27:37.637Z] 
[2021-07-05T12:27:37.637Z] C:\Users\jenkins\workspace\PR-26704-3-36c36af1-662d-42a5-bd57-41dc51718d7f\src\github.com\elastic\beats>go clean -modcache 
[2021-07-05T12:29:15.724Z] + gsutil --version
[2021-07-05T12:29:17.134Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-07-05T12:29:17.443Z] + gcloud auth activate-service-account --key-file ****
[2021-07-05T12:29:18.011Z] Activated service account credentials for: [[email protected]]
[2021-07-05T12:29:18.321Z] + gsutil -m -q cp -a public-read bWV0cmljYmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTBkM2NkZWJjMGJjNzk0NThiZDhhYzc5N2NmMTE4NmMwOTdkYjRlMTc2 gs://beats-ci-temp/ci/cache/
[2021-07-05T12:29:19.819Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-07-05T12:29:19.851Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-07-05T12:29:19.904Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26704/src/github.com/elastic/beats
[2021-07-05T12:29:20.940Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26704
[2021-07-05T12:29:20.986Z] [INFO] getVaultSecret: Getting secrets
[2021-07-05T12:29:21.023Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-05T12:29:21.698Z] + chmod 755 generate-build-data.sh
[2021-07-05T12:29:21.698Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26704/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26704/runs/3 FAILURE 2595428
[2021-07-05T12:29:21.698Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26704/runs/3/steps/?limit=10000 -o steps-info.json
[2021-07-05T12:29:29.723Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26704/runs/3/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 29379
Skipped 3892
Total 33271

@v1v
Copy link
Member Author

v1v commented Jul 5, 2021

/test

1 similar comment
@v1v
Copy link
Member Author

v1v commented Jul 5, 2021

/test

@v1v v1v merged commit c0341a8 into master Jul 5, 2021
@v1v v1v self-assigned this Jul 5, 2021
@v1v v1v deleted the revert-26617-feature/retry-option-prs-only branch July 5, 2021 13:24
v1v added a commit to v1v/beats that referenced this pull request Jul 5, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/retry-if-…
a26267c 
…stage-failed-within-same-build

* upstream/master: (36 commits)
  Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)" (elastic#26704)
  Packaging: linux/armv7 is not supported (elastic#26706)
  Cyberarkpas: Link to official docs on how to setup TLS (elastic#26614)
  Make network_direction, registered_domain and convert processors compatible with ES older than 7.13.0 (elastic#26676)
  Disable armv7 packaging (elastic#26679)
  [Heartbeat] use --params flag for synthetics (elastic#26674)
  Update dependent package to avoid downloading a suspicious file (elastic#26406)
  [mergify] set title and allow bp in any direction (elastic#26648)
  Fix memory leak in SQL helper when database is not available (elastic#26607)
  [CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)
  [mergify] automate PRs that change the backport rules (elastic#26641)
  [Metricbeat] Add Airflow module in xpack (elastic#26220)
  chore: add-backport-next (elastic#26620)
  [metricbeat] Add state_job metricset (elastic#26479)
  CI: jenkins labels are less time consuming now (elastic#26613)
  [MetricBeat] [AWS] Fix aws metric tags with resourcegroupstaggingapi paginator (elastic#26385) (elastic#26443)
  Move openmetrics module to oss (elastic#26561)
  Skip flaky test TestFilestreamMetadataUpdatedOnRename (elastic#26609)
  [filebeat][fortinet] Use default add_locale for fortinet.firewall (elastic#26524)
  Enroll proxy settings (elastic#26514)
  ...
v1v added a commit to v1v/beats that referenced this pull request Jul 8, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/use-debian-9
1d835f9 
* upstream/master: (430 commits)
  CI: increase timeout (elastic#26764)
  Heartbeat: add datastream fields to synthetics (elastic#26774)
  Osquerybeat: Change the query timeout from 3 secs to 60 secs (elastic#26775)
  Remove experimental warning for inputs with variables. (elastic#26762)
  Add latest k8s versions in testing (elastic#26729)
  change type of max_bytes to ByteType (elastic#26699)
  [Elastic Agent] Fix broken enrollment command (elastic#26749)
  Update agent managed manifest to include enrolment token variable (elastic#26756)
  Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (elastic#26737)
  Forward port changelog for 7.13.3 (elastic#26731) to master (elastic#26754)
  Upgrade PyYAML dependency used for tests (elastic#26746)
  Add agent fleet enrolment k8s manifest (elastic#26566)
  CI: retry the step only (elastic#26736)
  Osquerybeat: Fix the configuration poll interval setting (elastic#26739)
  [Filebeat] Replace copy_from with templated value (elastic#26631)
  Reduce the verbosity of the debug log for monitoring (elastic#26583)
  Add instructions on testing metricbeat kubernetes module (elastic#26643)
  Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)" (elastic#26704)
  Packaging: linux/armv7 is not supported (elastic#26706)
  Cyberarkpas: Link to official docs on how to setup TLS (elastic#26614)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cachedout cachedout cachedout approved these changes

Assignees

@v1v v1v

Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@v1v @elasticmachine @cachedout