[serve] Add Google Cloud Storage as a backend

[tkaymak]

Why are these changes needed?

Ray serve currently provides the ability to restore from a checkpoint in S3, this PR adds the possibility to use Google Cloud Storage (GCS) as a backend

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests

[simon-mo]

This is great! Can you also add "gcs" uri support to https://github.com/ray-project/ray/blob/master/python/ray/serve/storage/checkpoint_path.py#L20
and doc at
https://github.com/ray-project/ray/blob/master/doc/source/serve/deployment.rst#failure-recovery

[tkaymak]

Thank you @simon-mo - I've adjusted the code and the doc :)

[jiaodong]

Thanks for adding this support ! Can you also update your test plan in this PR with an example of running serve tests using GCS backend for checkpoint ? You can find one such example in https://sourcegraph.com/github.com/ray-project/ray/-/blob/release/serve_tests/workloads/serve_cluster_fault_tolerance.py?L54:59 that we kill the controller after initial deployment is done, then resume from checkpoint in external source. This test can be done using local ray cluster.

[simon-mo]

LGTM pending @jiaodong's comments. This is awesome!

[tkaymak]

@jiaodong I could, but only when I know a bucket name that you own and that the executor for that test has write access to that GCS bucket (so for example, GOOGLE_APPLICATION_CREDENTIALS is set for the test executor) - can you give me a bucket name?

[jiaodong]

@tkaymak that test i sent you is running on internal platform in particular so don't worry too much about writing to it :) What i was suggesting is a sanity e2e check using your own bucket on your laptop, then we can make internal changes needed to cover this case on CI afterwards.

[tkaymak]

@jiaodong - what do you think about this?

[jiaodong]

looks great, thanks for adding this ! I will follow up with changing bucket name afterwards.