Docs: We should have examples of allocate_empty_primary and allocate_stale_primary

[nik9000]

On this page we describe a few cluster reroute commands but we don't have examples of allocate_empty_primary and allocate_stale_primary. i know they are dangerous APIs that intentionally lose data, but if the disk that has the data on it is gone you want to run them and it'd be really nice to have a written out example on the page.

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-docs

[nik9000]

It turns out we've talked about this in the past and thought that it was safer not to have examples of these APIs because we really, really, really don't want folks copying and pasting them and losing data. Because these APIs lose data. That is the whole point. But my point stands: when you need them, you need them. And it is nice to have an example of the structure.

I wonder if we could document them with the allow_data_loss flag set to false so if anyone copies and pastes them then they won't work. I really just want them documented so you see the structure, and the structure is still there if you set that flag to false.

[vladimirdolzhenko]

It will appear indirectly due to work of another dangerous piece: #31389

it requires put a new allocation id and therefore provide command on how to allocate_stale_primary or in the worst scenario even allocate_empty_primary

[nik9000]

A small group of us talked about this and ended up pretty split but ultimately decided that the documentation that we have now is what we're going to stick with. The argument went like this:

We consider these command dangerous to copy and paste and execute.

But we consider these commands genuinely useful in certain circumstances: when you know you've lost data and there is no way to get it back, these commands put the index back into a state where you can write to them. Then you can work on getting the data base. If you have a recent backup then it is almost certainly better to restore the backup, flip an alias, and delete the broken index. But if you can't do that, then these commands are useful.

But we document how to send these command already. We just don't give examples. Knowing that you need these commands probably means that you are familiar enough with Elasticsearch to construct the command from the documentation.

But when you need these commands it means you've lost some data. Which probably means something catastrophic has happened and you are having a bad day. I know when I'm having a bad day I have a ton more trouble building commands from prose. Having an actual example will really help in these cases.

But when you are frustrated and having a bad day you are much more prone to making mistakes. This is exactly when you should slow down, read the prose a few times, then build the command. That makes triple sure that you understand the consequences of losing data.