[GCP] Adopt new provisioner to stop/down clusters

[Michaelvll]

This PR adopts the new provisioner refactoring from #2121.

Fixes #2089, to avoid the nodes to be leaked when the cluster is partially down.
Also Fixes #1983, as we will now send the requests first and wait the completion of together (using multiple threads in ray-project/ray#34455 is buggy, refer to #2160 (comment))

Previously:
Stopping 8 nodes takes: 168s

Now:
Stopping 8 nodes takes: 40s

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c test-gcp-mn --num-nodes 8 --cpus 2+ --cloud gcp; sky stop -y test-gcp-mn; sky down -y test-gcp-mn
  • sky launch -c test-gcp-mn --num-nodes 8 --cpus 2+ --cloud gcp; manually stop some of the nodes; sky down -y test-gcp-mn
  • sky launch -c test-gcp-tpu --cpus 2+ --gpus tpu-v2-8; sky stop test-gcp-tpu-mn; sky down -y test-gcp-tpu-mn
  • sky launch -c test-tpu-vm test.yaml; sky stop test-tpu-vm; sky start test-tpu-vm; sky down test-tpu-vm

    resources:
        accelerators: tpu-v2-8
        accelerator_args:
            tpu_vm: true
    

  • sky launch -c test-tpu-vm test.yaml; wait for the tpu vm to be preempted; sky status -r test-tpu-vm correctly terminate the tpu vm on the cloud.
• All smoke tests: pytest tests/test_smoke.py (except for test_cancel_gcp, test_distributed_tf due to the issue of the tensorflow code for resnet training [Examples] TF training code for ResNet no longer working #2207)
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[suquark]

why need to change this?

[Michaelvll]

This is a known issue that the 120 seconds is not enough for launching multiple nodes for GCP. We need to increase this to make sure sky launch --cloud gcp --num-nodes 8 to work.

[suquark]

LGTM

[Michaelvll]

Thanks for reviewing this @suquark! I am now testing it with:

• pytest tests/test_smoke.py (with [Core] Avoid install python 3.11 conda #2241 merged)