operators: ignore node deletion errors on absence

[burgerdev]

Context

If a node deletion operation fails transiently after the instance was deleted at the CSP, we need to ensure that the retry succeeds, even if the instance we attempt to delete is already gone at the CSP.

Example operator logs: https://gist.github.com/burgerdev/c80de86ae44552a383364ef031243a41

Proposed change(s)

• AWS: Ignore the specific error message for missing images.
• Azure: nothing - deleting absent instance IDs is no error here.
• GCP: turn on SkipInstancesOnValidationError, which is designed for this use case.

Additional info

• AB#3533

Checklist

• Run the E2E tests that are relevant to this PR's changes
  • upgrade test on GCP
  • upgrade test on AWS post-merge
• Add labels (e.g., for changelog category)
• Is PR title adequate for changelog?
• Link to Milestone

[netlify]

✅ Deploy Preview for constellation-docs canceled.

Name	Link
🔨 Latest commit	5949a5d
🔍 Latest deploy log	https://app.netlify.com/sites/constellation-docs/deploys/664cc07ce4e2070008434d91

[daniel-weisse]

Suggested change

	func isInstanceNotFoundError(err error) bool {
	if err == nil {
	return false
	}
	return strings.Contains(err.Error(), "Instance Id not found")
	}
	func isInstanceNotFoundError(err error) bool {
	var respErr *awshttp.ResponseError // using "awshttp" for package github.com/aws/aws-sdk-go-v2/aws/transport/http
	return errors.As(err, &respErr) && respErr.HTTPStatusCode() == http.StatusNotFound
	}

From https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling
Did not tests this, but if it works it would be preferable over string comparisons.

[burgerdev]

That would be preferable indeed, but AWS does not return 404s in that case, but 400s. I meant to also link to the logs but forgot - fixed that now (see above). I would not want to infer nonexistence from a 400, because that might hide other failure conditions, so I'm back at string comparison :/

<ErrorResponse xmlns="http://autoscaling.amazonaws.com/doc/2011-01-01/">
  <Error>
    <Type>Sender</Type>
    <Code>ValidationError</Code>
    <Message>Instance Id not found - No managed instance found for instance ID: i-061c63c5eb45f0416</Message>
  </Error>
  <RequestId>702f44ea-6d55-480b-8dfa-3da9e62f1634</RequestId>
</ErrorResponse>

[github-actions]

Coverage report

Package	Old	New	Trend
operators/constellation-node-operator/internal/cloud/aws/client	34.40%	34.60%	↗️
operators/constellation-node-operator/internal/cloud/gcp/client	43.80%	43.90%	↗️