gateway: unify root identifier resolution errors

[lidel]

Extracted from ipfs/kubo#9660 (review) so we dont forget

I was thinking about this again, both from semantic perspective, but also given your feedback on how hairy this gets.
I am leaning towards returning 500s for all invalid root identifiers, just for the sake of consistency and simplicity

@hacdias would like to hear your thoughts if below makes sense (if you dont have any concerns, feel free to execute on this):

• /ipfs/this-is-not-a-cid → we were thinking about 400 Bad Request, but it may be better/easier to return 500 Server Error, because maybe today we know this CID is not valid, but in 10 years the same software could be wrong? Maybe it is a valid CIDv8 that this old software can't recognize? 400 could be wrong, 500 will always kinda fit.
• /ipns/this-domain-does-not-exist → 500 (we dont know if domain does not exist, or was resolution error)
• /ipns/{cid-libp2p-key-that-cant-be-resolved} → 500 (if it is a valid key, resolution error, if not, we will fallbck to dnslink, and that also cant be proven negative)

TLDR

Return HTTP 500 Server Error for all /ip[nf]s/id failures.

TODO

• PR here: fix(gateway): return 500 for all /ip[nf]s/id failures #187
  • Side-quest: IsErrNotFound func probably should not have routing.ErrNotFound, because these errors are for /ipns/id, which is we always want to be 500.
• PR in Kubo and tests passing: feat(gateway): error handling improvements (500, 502, 504) kubo#9660
• PR against ipfs/specs, documenting this behavior Return 500 on resolution errors specs#380

[hacdias]

@lidel I completely agree. It's very hard to detect where these errors are coming from without changing a lot of libraries and ensuring we clearly bubble up the errors. I will work on the code to make all requests return HTTP 500 here.

[iand]

400 bad request is more appropriate for a CID that the server cannot parse since the server defines what request semantics it supports. The server hasn't failed and the user should not retry the request. A 500 response indicates to the user that they can retry it and expect it to work when the server has recovered.

[hacdias]

As I mentioned in Slack, I think we could keep /ipns the way it is and only revert to 400 in case there is an invalid CID under the /ipfs namespace. That may require some further changes but it is likely possible. However, that's also assuming a CID today and a CID in 10 years will look the same and/or be compatible.

[lidel]

@hacdias do you mind filling an issue for this so we tackle it after #176 ? (unless you've found an easy fix – maybe detect CID parsing errors in webError?)

I hoped this will be cosmetic, but this will be coming back, just like did for the opposite "why 400 on /ipns/, my domain is ok".
(We need to resolve this in both the go-libiofs/gateway library and ipfs/specs).