Persist quota information on disk during skyplane init

[simon-mo]

Closes #763

[sarahwooders]

I tried testing this by removing my ~/.skyplane folder and running skyplane init, but I got this error:

Error running _get_ec2_vm_quota: An error occurred (UnknownOperationException) when calling the GetServiceQuota
operation:
.... 
│ /Users/sarahwooders/repos/skyplane/skyplane/compute/aws/aws_auth.py:42 in _get_ec2_vm_quota      │
│                                                                                                  │
│    39 │   │                                                                                      │
│    40 │   │   name_to_quota = {}                                                                 │
│    41 │   │   for name, code in [("on_demand_standard_vcpus", "L-1216C47A"), ("spot_standard_v   │
│ ❱  42 │   │   │   retrieved_quota = quotas_client.get_service_quota(ServiceCode="ec2", QuotaCo   │
│    43 │   │   │   name_to_quota[name] = int(retrieved_quota["Quota"]["Value"])                   │
│    44 │   │   return name_to_quota                                                               │
│    45                                                                                            │
│                                                                                                  │
│ ╭────────────────────────────────────────── locals ──────────────────────────────────────────╮   │
│ │          code = 'L-1216C47A'                                                               │   │
│ │          name = 'on_demand_standard_vcpus'                                                 │   │
│ │ name_to_quota = {}                                                                         │   │
│ │ quotas_client = <botocore.client.ServiceQuotas object at 0x7f81f97850d0>                   │   │
│ │        region = 'eu-central-2'                                                             │   │
│ │          self = <skyplane.compute.aws.aws_auth.AWSAuthentication object at 0x7f820fa721f0> │   │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────╯   │
│                                                                                                  │
│ /Users/sarahwooders/repos/skyplane/env/lib/python3.9/site-packages/botocore/client.py:514 in     │
│ _api_call                                                                                        │
│                                                                                                  │
│    511 │   │   │   │   │   f"{py_operation_name}() only accepts keyword arguments."              │
│    512 │   │   │   │   )                                                                         │
│    513 │   │   │   # The "self" in this scope is referring to the BaseClient.                    │
│ ❱  514 │   │   │   return self._make_api_call(operation_name, kwargs)                            │
│    515 │   │                                                                                     │
│    516 │   │   _api_call.__name__ = str(py_operation_name)                                       │
│    517                                                                                           │
│                                                                                                  │
│ ╭─────────────────────────────────── locals ───────────────────────────────────╮                 │
│ │              args = ()                                                       │                 │
│ │            kwargs = {'ServiceCode': 'ec2', 'QuotaCode': 'L-1216C47A'}        │                 │
│ │    operation_name = 'GetServiceQuota'                                        │                 │
│ │ py_operation_name = 'get_service_quota'                                      │                 │
│ │              self = <botocore.client.ServiceQuotas object at 0x7f81f97850d0> │                 │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                 │
│                                                                                                  │
│ /Users/sarahwooders/repos/skyplane/env/lib/python3.9/site-packages/botocore/client.py:938 in     │
│ _make_api_call                                                                                   │
│                                                                                                  │
│    935 │   │   if http.status_code >= 300:                                                       │
│    936 │   │   │   error_code = parsed_response.get("Error", {}).get("Code")                     │
│    937 │   │   │   error_class = self.exceptions.from_code(error_code)                           │
│ ❱  938 │   │   │   raise error_class(parsed_response, operation_name)                            │
│    939 │   │   else:                                                                             │
│    940 │   │   │   return parsed_response                                                        │
│    941                                                                                           │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │      api_params = {'ServiceCode': 'ec2', 'QuotaCode': 'L-1216C47A'}                          │ │
│ │     error_class = <class 'botocore.exceptions.ClientError'>                                  │ │
│ │      error_code = 'UnknownOperationException'                                                │ │
│ │  event_response = None                                                                       │ │
│ │         handler = <function inject_api_version_header_if_needed at 0x7f820f9cadc0>           │ │
│ │            http = <botocore.awsrequest.AWSResponse object at 0x7f81f9a46190>                 │ │
│ │ operation_model = OperationModel(name=GetServiceQuota)                                       │ │
│ │  operation_name = 'GetServiceQuota'                                                          │ │
│ │ parsed_response = {                                                                          │ │
│ │                   │   'Error': {'Message': '', 'Code': 'UnknownOperationException'},         │ │
│ │                   │   'ResponseMetadata': {                                                  │ │
│ │                   │   │   'RequestId': '3c58a495-1950-4954-8415-522dc684970e',               │ │
│ │                   │   │   'HTTPStatusCode': 400,                                             │ │
│ │                   │   │   'HTTPHeaders': {                                                   │ │
│ │                   │   │   │   'date': 'Wed, 08 Mar 2023 04:56:48 GMT',                       │ │
│ │                   │   │   │   'content-type': 'application/x-amz-json-1.1',                  │ │
│ │                   │   │   │   'content-length': '38',                                        │ │
│ │                   │   │   │   'connection': 'keep-alive',                                    │ │
│ │                   │   │   │   'x-amzn-requestid': '3c58a495-1950-4954-8415-522dc684970e'     │ │
│ │                   │   │   },                                                                 │ │
│ │                   │   │   'RetryAttempts': 0                                                 │ │
│ │                   │   }                                                                      │ │
│ │                   }                                                                          │ │
│ │ request_context = {                                                                          │ │
│ │                   │   'client_region': 'eu-central-2',                                       │ │
│ │                   │   'client_config': <botocore.config.Config object at 0x7f81f9785250>,    │ │
│ │                   │   'has_streaming_input': False,                                          │ │
│ │                   │   'auth_type': None,                                                     │ │
│ │                   │   'retries': {                                                           │ │
│ │                   │   │   'attempt': 1,                                                      │ │
│ │                   │   │   'invocation-id': '8b5b59f1-e355-4cf8-b161-3625a237d520',           │ │
│ │                   │   │   'max': 5                                                           │ │
│ │                   │   },                                                                     │ │
│ │                   │   'timestamp': '20230308T045647Z'                                        │ │
│ │                   }                                                                          │ │
│ │    request_dict = {                                                                          │ │
│ │                   │   'url_path': '/',                                                       │ │
│ │                   │   'query_string': '',                                                    │ │
│ │                   │   'method': 'POST',                                                      │ │
│ │                   │   'headers': {                                                           │ │
│ │                   │   │   'X-Amz-Target': 'ServiceQuotasV20190624.GetServiceQuota',          │ │
│ │                   │   │   'Content-Type': 'application/x-amz-json-1.1',                      │ │
│ │                   │   │   'User-Agent': 'Boto3/1.24.59 Python/3.9.12 Darwin/21.3.0           │ │
│ │                   Botocore/1.27.96'                                                          │ │
│ │                   │   },                                                                     │ │
│ │                   │   'body': b'{"ServiceCode": "ec2", "QuotaCode": "L-1216C47A"}',          │ │
│ │                   │   'url': 'https://servicequotas.eu-central-2.amazonaws.com/',            │ │
│ │                   │   'context': {                                                           │ │
│ │                   │   │   'client_region': 'eu-central-2',                                   │ │
│ │                   │   │   'client_config': <botocore.config.Config object at                 │ │
│ │                   0x7f81f9785250>,                                                           │ │
│ │                   │   │   'has_streaming_input': False,                                      │ │
│ │                   │   │   'auth_type': None,                                                 │ │
│ │                   │   │   'retries': {                                                       │ │
│ │                   │   │   │   'attempt': 1,                                                  │ │
│ │                   │   │   │   'invocation-id': '8b5b59f1-e355-4cf8-b161-3625a237d520',       │ │
│ │                   │   │   │   'max': 5                                                       │ │
│ │                   │   │   },                                                                 │ │
│ │                   │   │   'timestamp': '20230308T045647Z'                                    │ │
│ │                   │   }                                                                      │ │
│ │                   }                                                                          │ │
│ │            self = <botocore.client.ServiceQuotas object at 0x7f81f97850d0>                   │ │
│ │      service_id = 'service-quotas'                                                           │ │
│ │    service_name = 'service-quotas'                                                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ClientError: An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:

[sarahwooders]

For Azure-only, I get this error even after running pip install skyplane[azure]:

│ /Users/sarahwooders/repos/skyplane/skyplane/compute/azure/azure_auth.py:71 in save_region_config │
│                                                                                                  │
│    68 │   │   │   f.write("\n".join(region_list))                                                │
│    69 │   │                                                                                      │
│    70 │   │   # Get Quotas                                                                       │
│ ❱  71 │   │   quota_client = self.get_quota_client()                                             │
│    72 │   │                                                                                      │
│    73 │   │   def get_quota(region):                                                             │
│    74 │   │   │   try:                                                                           │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │                 f = <_io.TextIOWrapper name='/Users/sarahwooders/.skyplane/azure_config'     │ │
│ │                     mode='w' encoding='UTF-8'>                                               │ │
│ │ HttpResponseError = <class 'azure.core.exceptions.HttpResponseError'>                        │ │
│ │          location = <azure.mgmt.subscription.models._models_py3.Location object at           │ │
│ │                     0x7f8f7ac03610>                                                          │ │
│ │       region_list = []                                                                       │ │
│ │              self = <skyplane.compute.azure.azure_auth.AzureAuthentication object at         │ │
│ │                     0x7f8f79ea2f40>                                                          │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/sarahwooders/repos/skyplane/skyplane/utils/imports.py:30 in wrapped                       │
│                                                                                                  │
│   27 │   │   │   │   │   │   │   module_imported = importlib.import_module(".".join(module_sp    │
│   28 │   │   │   │   │   │   │   modules_imported.append(getattr(module_imported, module_spli    │
│   29 │   │   │   │   │   │   except (ImportError, AttributeError):                               │
│ ❱ 30 │   │   │   │   │   │   │   raise ImportError(err_msg) from e                               │
│   31 │   │   │   │   │   else:                                                                   │
│   32 │   │   │   │   │   │   raise ImportError(err_msg) from e                                   │
│   33 │   │   │   return fn(*modules_imported, *args, **kwargs)                                   │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │             args = (                                                                         │ │
│ │                    │   <skyplane.compute.azure.azure_auth.AzureAuthentication object at      │ │
│ │                    0x7f8f79ea2f40>,                                                          │ │
│ │                    )                                                                         │ │
│ │          err_msg = 'Cannot import azure.mgmt.quota.AzureQuotaExtensionAPI. Install skyplane  │ │
│ │                    with azu'+43                                                              │ │
│ │               fn = <function AzureAuthentication.get_quota_client at 0x7f8f78ba3040>         │ │
│ │           kwargs = {}                                                                        │ │
│ │           module = 'azure.mgmt.quota.AzureQuotaExtensionAPI'                                 │ │
│ │     module_split = ['azure', 'mgmt', 'quota', 'AzureQuotaExtensionAPI']                      │ │
│ │          modules = ('azure.mgmt.quota.AzureQuotaExtensionAPI',)                              │ │
│ │ modules_imported = []                                                                        │ │
│ │        pip_extra = 'azure'                                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: Cannot import azure.mgmt.quota.AzureQuotaExtensionAPI. Install skyplane with azure support: `pip install
'skyplane[azure]'`

[simon-mo]

The azure issue require you to install the dep manually for now because it hasn’t been release yet

[simon-mo]

For aws, what’s your account’s iam capabilities? Looks like I need to catch and issue a warning here. It works on admin account.

[sarahwooders]

I have root credentials for AWS. It seems to only fail on specific regions:

eu-south-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:
eu-central-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:
ap-south-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:
eu-south-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:
eu-central-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:
ap-south-2 An error occurred (UnknownOperationException) when calling the GetServiceQuota operation:

Not sure what the reason is, but we can maybe just set them to zero.

I got Azure to run, but it wrote an empty map in azure_quota.

[sarahwooders]

Could you also fix the failing unit tests?

[simon-mo]

Ah probably disabled regions.. I'll add some error handling.
The issue with azure could be the quota api was enabled asynchronously so it might even take a minute or two so we can query the api. Originally I thought the time it takes waiting for other resources should be fine but it appears not.