[BUG]: Credentials error when installing UCX ("Unable to load AWS credentials from any provider in the chain")

[JuanDiego18]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The customer is having a credentials issue while trying to install UCX. They have tried a couple of things but still getting the same issue. They tried installing UCX, running the upgrade command and manually deleting the workflows in the workspace and then running the install command. Getting the same error in all scenarios. Even installing UCX in a new workspace failed with the same error.

Expected Behavior

UCX installs properly

Steps To Reproduce

No response

Cloud

AWS

Operating System

Linux

Version

latest via Databricks CLI

Relevant log output

ERROR [d.l.blueprint.parallel][installing_components_0] installing components task failed: org.apache.hadoop.hive.ql.metadata.HiveException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@51650af3: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4a4831f7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]: Traceback (most recent call last):

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner

    return func(*args, **kwargs), None

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 510, in _create_database

    raise err

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 499, in _create_database

    deploy_schema(self._sql_backend, self._config.inventory_database)

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 85, in deploy_schema

    deployer.deploy_schema()

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema

    self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}")

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute

    self._sql.execute(sql, catalog=catalog, schema=schema)

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute

    self._raise_if_needed(result_status)

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed

    raise error_class(error_message)

databricks.sdk.errors.platform.BadRequest: org.apache.hadoop.hive.ql.metadata.HiveException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@51650af3: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4a4831f7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

14:56:04 ERROR [d.l.blueprint.parallel] More than half 'installing components' tasks failed: 0% results available (0/2). Took 0:03:09.057741

Traceback (most recent call last):

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 830, in <module>

    workspace_installer.run()

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 195, in run

    workspace_installation.run()

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 486, in run

    Threads.strict("installing components", install_tasks)

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 62, in strict

    raise errs[0]

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 158, in inner

    return func(*args, **kwargs), None

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 510, in _create_database

    raise err

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 499, in _create_database

    deploy_schema(self._sql_backend, self._config.inventory_database)

  File "/home/grobert5/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 85, in deploy_schema

    deployer.deploy_schema()

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/deployment.py", line 18, in deploy_schema

    self._sql_backend.execute(f"CREATE SCHEMA IF NOT EXISTS hive_metastore.{self._inventory_schema}")

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/backends.py", line 142, in execute

    self._sql.execute(sql, catalog=catalog, schema=schema)

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 265, in execute

    self._raise_if_needed(result_status)

  File "/home/grobert5/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 479, in _raise_if_needed

    raise error_class(error_message)

databricks.sdk.errors.platform.BadRequest: org.apache.hadoop.hive.ql.metadata.HiveException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@51650af3: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4a4831f7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

Error: installer: exit status 1

[asnare]

@JuanDiego18: This looks like an AWS authentication issue. Have you set the AWS CLI up and logged in? What does aws sts get-caller-identity say?

[JuanDiego18]

Hi, @asnare ; let me check that and get back to you. Having said that, why would we need to authenticate to the AWS CLI for UCX installation?

[JuanDiego18]

Hi @asnare @HariGS-DB , this is what aws sts get-caller-identity says:

{
"Userld": "<redacted>",
"Account": "<redacted>",
"Arn": "arn:aws:sts::<redacted>:assumed-role/<redacted>"
}