Retry "non retryable" error on worker long poll

[Quinn-With-Two-Ns]

Retry "non retryable" errors for a period of time like core. This helps handle some know edge cases where proxies may way rewrite grpc timeouts when a server fails over and cause an error from the server. This aligns the go sdk with core. The retry period was picked at 2 minutes because it is greater than the long poll interval so in the proxy failure mode we still retry at least once.

See also:
temporalio/features#218

[cretz]

Looks perfect!

[cretz]

@Quinn-With-Two-Ns - Is the increase in timeout needed because of this change? I'm not sure our integration test for worker error should have to wait a whole 2 minutes when it took seconds before. Maybe it'd be worth a way to turn this 2m minimum off and if we want to test the full time taken before fatal, we can move that to features repo.

[Quinn-With-Two-Ns]

@cretz Yes that is why it is needed, Moving to features does not help because we run features test as part of CI.

[Quinn-With-Two-Ns]

I looked into running these tests in parallel, but testify does not support it.

We could lower the timeout for test, but that would require exposing the timeout to users, which I thought we didn't want to do.

[cretz]

We could lower the timeout for test, but that would require exposing the timeout to users, which I thought we didn't want to do.

I agree we probably don't want to do it. Technically you can access internal from our integration tests, but I'd be ok with exposing as a worker option too. If the choice is between exposing and adding 2 mins to every test run, I think the former is better :-) (but again, if you want can just make a visible var in internal I think)