Add data stream support to Snapshot Restore SLM policies

[matt-davis-elastic]

Why should we add this?

Elasticsearch is adding a new primitive called data streams. Everywhere in our UI's where a user could reference/use an Index we should allow a user to reference a data stream.

What should be changed?

Looking in the SLM UI I see that during the creation of a policy you have the ability to specify an index pattern. This should show data streams in the dropdown.

Matt K is needing similar information for the Kibana Index Patterns UI to see that ticket look here. I wonder if the same API call/calls could be used here as well.

Similarly, when you go to restore a snapshot there should be data streams in the restore dropdown.

I am not sure how the snapshot API will change to reflect data streams. I will update the ticket here once I have more information.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[matt-davis-elastic]

@jethr0null @cjcenizal Let me know if I overlooked anything.

[cjcenizal]

@matt-davis-elastic Do you know if there's a use case for people to back up a subset of a data stream's backing indices?

[cjcenizal]

Proposal

Clear distinction between indices and data streams. We'll create a separate list for selecting the data streams you want to back up. We will prevent backing indices from being listed in the indices list.

Other API options. The current API has an ignore_unavailable option for ignoring unavailable indices. Will this option be extended to cover data streams as well? Will we add a separate option for data streams? Will we add any other options?

[cjcenizal]

@martijnvg Could you take a look at my questions under the "Other API options" section, above?

[matt-davis-elastic]

@cjcenizal I am not currently aware of a use case where someone would want to backup a subset of the data stream. With that said I could easily see someone coming up with a good use case and asking to do it. I think if it's not too much work, and it doesn't make the UX clumsy, we should think about supporting it. With that said we should put some verbiage around it. If you backup only a subset it's my understanding that they won't belong to a data set.

[matt-davis-elastic]

@cjcenizal Looking at the mockups I was thinking it might be better to try to unify the UX with what the Index Patterns were going to do. So my thoughts were that we would have one UI element that would cover both indices and data streams. Also maybe it makes sense to add aliases in as well. Thoughts?

[martijnvg]

So my thoughts were that we would have one UI element that would cover both indices and data streams.

I think we should threat data streams and indices as one of the things that can be snapshotted. Therefor I think we treat them the same in the ui and snapshot apis. If not snapshotting everything then some can specify data stream names or index/alias names.

When it comes to 'Other API options', I think we shouldn't add data stream specific options here. If a data stream is not available or some of the backing indices or not available then depending whether ignore_unavailable is toggled it is not snapshotted or partially snapshotted.

[cjcenizal]

@matt-davis-elastic Thanks for the info and suggestions! We will update the UI to support selection of a subset of backing indices.

Can you help me understand a user's mindset when they're backing up data from the perspective of index aliases? Are you thinking that a user might be so used to identifying their data set by the index alias, that they'll want to "select" this index alias in order to back up the index behind it? Or are you thinking that it's more useful for providing context, so that a user looking at a list of indices and trying to make a selection would be able to choose an index with more confidence if they can also see the aliases with which its associated?

@martijnvg Good points. Everything you said makes sense to me from the API side of things. It sounds like you're proposing that we don't make any changes to the API? If so then this will make it easier for us on the UI side.

[matt-davis-elastic]

@cjcenizal I was thinking that if we are showing data streams and the indices that sit behind them it might make sense visually to show the same type of view for aliases. I think it could help with confidence as well.

[cjcenizal]

Addressed by #68078