Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[esArchiver] retry delete/create for non-kibana indexes which conflict with aliases #83746

Closed
wants to merge 1 commit into from
Closed

[esArchiver] retry delete/create for non-kibana indexes which conflict with aliases #83746

wants to merge 1 commit into from

Conversation

spalger
Copy link
Contributor

@spalger spalger commented Nov 18, 2020

In #71910 I encoded an assumption that alias conflicts would always be caused by .kibana indexes, but there are other indexes which might be automatically created and pointing at an alias, like the .kibana_task_manager indexes. In the rare case that one of those indices is recreated in conflict with the index load being attempted by the EsArchiver we log a warning and then retry the create function, assuming that the create function will clear the index like we do for .kibana_\d+ indexes. Instead, the clearing logic is skipped because .kibana_task_manager doesn't match the pattern and we just keep retrying the create without improving our chances of success.

This updates the logic to use the isKibanaIndex test from the kibana_index.ts module everywhere, so that we're no longer treating documents for .kibana_task_manager differently in different places of the code, and also ensures that we delete the .kibana_* indexes which aren't saved object indexes on conflicts.

@kibanamachine
Copy link
Contributor

kibanamachine commented Nov 19, 2020 •
edited
Loading

💔 Build Failed

Failed CI Steps

Test Failures

Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/security/rbac_phase1·js.security app rbac "before all" hook for "rbac all role can save a visualization"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: security app
[00:00:00]           └-> "before all" hook
[00:06:50]           └-: rbac 
[00:06:50]             └-> "before all" hook
[00:06:50]             └-> "before all" hook
[00:06:50]               │ debg users
[00:06:50]               │ info [logstash_functional] Loading "mappings.json"
[00:06:50]               │ info [logstash_functional] Loading "data.json.gz"
[00:06:50]               │ info Taking screenshot "/dev/shm/workspace/parallel/14/kibana/x-pack/test/functional/screenshots/failure/security app rbac  _before all_ hook.png"
[00:06:50]               │ info Current URL is: http://localhost:61141/login
[00:06:50]               │ info Saving page source to: /dev/shm/workspace/parallel/14/kibana/x-pack/test/functional/failure_debug/html/security app rbac  _before all_ hook.html
[00:06:50]               └- ✖ fail: security app rbac  "before all" hook for "rbac all role can save a visualization"
[00:06:50]               │      Error: [resource_already_exists_exception] index [logstash-2015.09.22/qS7JFlc9Sg2W3s3xSQPVRw] already exists, with { index_uuid="qS7JFlc9Sg2W3s3xSQPVRw" & index="logstash-2015.09.22" }
[00:06:50]               │       at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
[00:06:50]               │       at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
[00:06:50]               │       at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
[00:06:50]               │       at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
[00:06:50]               │       at endReadableNT (_stream_readable.js:1223:12)
[00:06:50]               │       at processTicksAndRejections (internal/process/task_queues.js:84:21)
[00:06:50]               │ 
[00:06:50]               │

Stack Trace

StatusCodeError: [resource_already_exists_exception] index [logstash-2015.09.22/qS7JFlc9Sg2W3s3xSQPVRw] already exists, with { index_uuid="qS7JFlc9Sg2W3s3xSQPVRw" & index="logstash-2015.09.22" }
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/logstash-2015.09.22',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'resource_already_exists_exception',
      reason: 'index [logstash-2015.09.22/qS7JFlc9Sg2W3s3xSQPVRw] already exists',
      index_uuid: 'qS7JFlc9Sg2W3s3xSQPVRw',
      index: 'logstash-2015.09.22'
    },
    status: 400
  },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [logstash-2015.09.22/qS7JFlc9Sg2W3s3xSQPVRw] already exists","index_uuid":"qS7JFlc9Sg2W3s3xSQPVRw","index":"logstash-2015.09.22"}],"type":"resource_already_exists_exception","reason":"index [logstash-2015.09.22/qS7JFlc9Sg2W3s3xSQPVRw] already exists","index_uuid":"qS7JFlc9Sg2W3s3xSQPVRw","index":"logstash-2015.09.22"},"status":400}',
  toString: [Function],
  toJSON: [Function]
}
Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/uptime.Uptime app with generated data "before each" hook: load heartbeat data for "displays the overall availability"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Uptime app
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: with generated data
[00:00:00]             └-> "before all" hook

Stack Trace

StatusCodeError: [resource_already_exists_exception] index [heartbeat-8-generated-test/bCjxXmB3QK-mEiy_Nz9p4w] already exists, with { index_uuid="bCjxXmB3QK-mEiy_Nz9p4w" & index="heartbeat-8-generated-test" }
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/heartbeat-8-generated-test',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'resource_already_exists_exception',
      reason: 'index [heartbeat-8-generated-test/bCjxXmB3QK-mEiy_Nz9p4w] already exists',
      index_uuid: 'bCjxXmB3QK-mEiy_Nz9p4w',
      index: 'heartbeat-8-generated-test'
    },
    status: 400
  },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [heartbeat-8-generated-test/bCjxXmB3QK-mEiy_Nz9p4w] already exists","index_uuid":"bCjxXmB3QK-mEiy_Nz9p4w","index":"heartbeat-8-generated-test"}],"type":"resource_already_exists_exception","reason":"index [heartbeat-8-generated-test/bCjxXmB3QK-mEiy_Nz9p4w] already exists","index_uuid":"bCjxXmB3QK-mEiy_Nz9p4w","index":"heartbeat-8-generated-test"},"status":400}',
  toString: [Function],
  toJSON: [Function]
}
X-Pack Accessibility Tests.x-pack/test/accessibility/apps/roles·ts.Kibana roles page a11y tests "before all" hook for "a11y test for Roles main page"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:04:31]         └-: Kibana roles page a11y tests
[00:04:31]           └-> "before all" hook
[00:04:31]           └-> "before all" hook
[00:04:31]             │ info [logstash_functional] Loading "mappings.json"
[00:04:31]             │ info [logstash_functional] Loading "data.json.gz"
[00:04:31]             │ info Taking screenshot "/dev/shm/workspace/parallel/18/kibana/x-pack/test/functional/screenshots/failure/Kibana roles page a11y tests _before all_ hook.png"
[00:04:31]             │ info Current URL is: http://localhost:61181/app/management/security/users/edit/deleteA11y
[00:04:31]             │ info Saving page source to: /dev/shm/workspace/parallel/18/kibana/x-pack/test/functional/failure_debug/html/Kibana roles page a11y tests _before all_ hook.html
[00:04:31]             └- ✖ fail: Kibana roles page a11y tests "before all" hook for "a11y test for Roles main page"
[00:04:31]             │      Error: [resource_already_exists_exception] index [logstash-2015.09.22/ZWtWC__OT8WNXR2pe4EW3w] already exists, with { index_uuid="ZWtWC__OT8WNXR2pe4EW3w" & index="logstash-2015.09.22" }
[00:04:31]             │       at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
[00:04:31]             │       at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
[00:04:31]             │       at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
[00:04:31]             │       at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
[00:04:31]             │       at endReadableNT (_stream_readable.js:1223:12)
[00:04:31]             │       at processTicksAndRejections (internal/process/task_queues.js:84:21)
[00:04:31]             │ 
[00:04:31]             │

Stack Trace

StatusCodeError: [resource_already_exists_exception] index [logstash-2015.09.22/ZWtWC__OT8WNXR2pe4EW3w] already exists, with { index_uuid="ZWtWC__OT8WNXR2pe4EW3w" & index="logstash-2015.09.22" }
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/logstash-2015.09.22',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'resource_already_exists_exception',
      reason: 'index [logstash-2015.09.22/ZWtWC__OT8WNXR2pe4EW3w] already exists',
      index_uuid: 'ZWtWC__OT8WNXR2pe4EW3w',
      index: 'logstash-2015.09.22'
    },
    status: 400
  },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [logstash-2015.09.22/ZWtWC__OT8WNXR2pe4EW3w] already exists","index_uuid":"ZWtWC__OT8WNXR2pe4EW3w","index":"logstash-2015.09.22"}],"type":"resource_already_exists_exception","reason":"index [logstash-2015.09.22/ZWtWC__OT8WNXR2pe4EW3w] already exists","index_uuid":"ZWtWC__OT8WNXR2pe4EW3w","index":"logstash-2015.09.22"},"status":400}',
  toString: [Function],
  toJSON: [Function]
}

and 11 more failures, only showing the first 3.

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spalger spalger closed this Dec 2, 2020
@spalger spalger deleted the fix/es-archiver-retry branch January 7, 2021 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@spalger @kibanamachine