Don't fail if schema_version and schema_update_history tables already exist

[markmark206]

What changed?

Don't fail if cassandra tables already exist. This allows us to retry and roll forward in case of mid-stream failures during installations.

Why?

Our test pipelines occasionally experienced failures, especially as we started testing with larger, replicated cassandra cluster

How did you test it?
I built a docker image with this change and used it to deploy to kubernetes cluster a few times.

Potential risks

If this doesn't work for some reason, we can rollback or fix.

[underrun]

this looks okay to me but are there any scenarios where failing here is a safety feature for users? if not cool - if so then maybe we can verify cassandra is fully ready before creating these at all?

[markmark206]

this looks okay to me but are there any scenarios where failing here is a safety feature for users? if not cool - if so then maybe we can verify cassandra is fully ready before creating these at all?

Great point, thank you! I am not aware of any, but I am curious if anyone has any additional insights here.

(The intention here is to address the failure scenario outlined in the side note in temporalio/helm-charts#65

I am somewhat reluctant to gate the creation logic on an external "all #{replication_factor} replicas are online" check, since things could change between that check and executing this logic, in which case we end up in the same current non-recoverable failure state, just like we did before.)

[shawnhathaway]

this looks okay to me but are there any scenarios where failing here is a safety feature for users? if not cool - if so then maybe we can verify cassandra is fully ready before creating these at all?

Ideally all persistence queries are idempotent and we craft non-idempotent queries when we need them