Investigate if ArtifactsSuite/TestArtifactGC is flakey

[isubasinghe]

No description provided.

[juliev0]

Is it possible to provide a link to the failed run?

[isubasinghe]

@juliev0 I believe it is this one: https://github.com/argoproj/argo-workflows/actions/runs/3691130662/jobs/6248880760

[juliev0]

@juliev0 I believe it is this one: https://github.com/argoproj/argo-workflows/actions/runs/3691130662/jobs/6248880760

Thanks for finding it!

[juliev0]

For some reason the wait container isn't saving the artifact into minio some of the time, so then when the test expects that the artifact was not deleted and looks for it, it was never saved in the first place, so it fails. The wait container is apparently exiting with exit code 2, but there's no error displayed.

Attaching result of "kubectl describe pod" as well as wait container log:
describe-pod.log
wait.log

[juliev0]

It seems like maybe the wait container is dying before it's finished, according to this sequence of events:

1. Here the Controller determines that it can send a SIGTERM to the Pod only once the main container is done (but the wait container may still be running) (I can see in the log that this is happening)
2. The wait container has tied its context to SIGTERM, such that the context is marked 'done' when the SIGNAL arrives here.
3. Somehow the wait container seems to be issuing an exit code 2 when the SIGTERM arrives. I'm not sure what would cause an Exit Code 2?