[Breaking change] Drop support for copy/import conflicts ("pseudo copies")

[jportner]

Change description

Which release will ship the breaking change?

8.0

Describe the change. How will it manifest to users?

Starting in 7.10, when copying or importing saved objects, users have two options:

"Create new objects with random IDs" is a new option. Starting in 7.11, this will be selected by default.

Starting in 8.0, "Create new objects with random IDs" will be the only option (users will no longer see these radio controls at all).

How many users will be affected?

Telemetry: TBD #81907 <-- BLOCKER

What can users do to address the change manually?

• Import: Instead of using imports as "pseudo snapshots" by overwriting existing objects, users can take advantage of Kibana snapshots.
• Copy: Instead of using copies for "pseudo sharing" by overwriting existing objects, users can take advantage of saved object sharing.
  • Note: This will only be possible if all saved object types have been converted to multi-namespace types by the 8.0 release!

How could we make migration easier with the Upgrade Assistant?

N/A

Are there any edge cases?

None that come to mind.

Test Data

Provide test data. We can’t build a solution without data to test it against.

Cross links

Cross-link to relevant Elasticsearch breaking changes.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[rudolf]

I think the reason users use import/export for "psuedo snapshot" is because they rely on the ability to do a partial snapshot restore. While you could restore individual documents from an Elasticsearch snapshot there is no UI for it and because you have to follow all of a document's references it's complex enough for it to be outside of the reach of most users.

[cjcenizal]

Instead of using imports as "pseudo snapshots" by overwriting existing objects, users can take advantage of Kibana snapshots.

I think the reason users use import/export for "pseudo snapshot" is because they rely on the ability to do a partial snapshot restore. While you could restore individual documents from an Elasticsearch snapshot there is no UI for it and because you have to follow all of a document's references it's complex enough for it to be outside of the reach of most users.

To clarify, it sounds like the original guidance of taking snapshots of saved objects and then restoring those snapshots to create saved object state will be a degraded UX from the current functionality of importing a saved objects file, because users will lose the ability to easily select the saved objects they want to restore. Is that right?

If so, then maybe we should add a note that this deprecation is blocked on shoring up that gap in the UX? I'm not fully aware of the tradeoffs being considered here, but I'm wary of making features more difficult to use.

In terms of improving the UX of restoring saved objects from a snapshot, I think we'd need to enlist the help of a designer to explore what this would look like and how it would behave. Based on the outcome we might then need to ask the ES engineers for an API to let us list the saved object documents in the backup, though searchable snapshots might provide this for us already.

[rudolf]

To clarify, it sounds like the original guidance of taking snapshots of saved objects and then restoring those snapshots to create saved object state will be a degraded UX from the current functionality of importing a saved objects file, because users will lose the ability to easily select the saved objects they want to restore. Is that right?

Yes. This will definitely remove existing functionality that we know users are currently relying on and as such needs to be carefully considered. We don't know how many users use this functionality, so #81907 could shed some light on the impact. I think that's the intention, but @jportner could we update the issue to make it clear that this is blocked blocked until we can collect more telemetry?

Given the high number of upvotes on the original import/export issue #1552, my gut feeling is that we won't be able to remove this without a replacement.

[rudolf]

This could also potentially impact #2310 (use-case is explained in more detail in elastic/cloud-on-k8s#3598).

[alisonelizabeth]

I'm going to remove the Elasticsearch UI team label. This deprecation should be registered by the plugin owner via the core deprecations service (#94845). All registered deprecations will be displayed in the Upgrade Assistant (to be implemented via #97159). Feel free to reach out to myself or the core team with any questions!

[jportner]

Quick recap

Pseudo-copy mode (the "Check for existing objects" option) and true copy mode (the "Create new objects with random IDs" option) are mutually exclusive options in both the "Import saved objects" feature and the "Copy saved objects to space" feature.

The motivation behind removing pseudo-copy mode is so we can remove the underlying code and decrease our technical debt. However, all of that code actually resides in the Import feature (the Copy feature relies on it). So it doesn't make a whole lot of sense to try to remove pseudo-copy mode from the Copy feature until we can also remove it from the Import feature.

We have decided that we cannot remove pseudo-copy mode until:

1. Users have a reasonable alternative to keep objects in sync across spaces (sharing saved objects)
2. Users have a reasonable alternative to create and restore snapshots within Kibana

Currently the defaults are:

• Import feature: pseudo-copy mode
• Copy feature: true copy mode

Current usage data (past 90 days) indicates:

• Import feature: 99.7% pseudo-copy mode, 0.3% true copy mode
• Copy feature: 3% pseudo-copy mode, 97% true copy mode

Sharing saved objects

At this point in time, it appears that most object types will not be shareable until after the 8.0 release. We don't have a meta-issue for actually making all objects shareable yet, but we do have a meta-issue for making them share-capable: ##100489. After all objects are actually shareable, we will also need some lead time after that to collect more usage data and determine whether users are still relying on pseudo-copy. We may need to provide some additional functionality (such as "merging" alike objects) to steer users towards using sharing.

Snapshots in Kibana

We don't have an issue for this yet but it has come up in discussion. Users are currently relying on export/import with pseudo-copy functionality as a way to create and restore "snapshots". The thinking goes that if we supported true snapshots of the .kibana index within the Kibana UI, this is a better alternative.

Somewhat related: #91615

Closing this issue for now as we don't have a realistic estimate as to when we can actually drop support for pseudo-copy mode.