Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AZ ML CLI login issue with MSI #2461

Closed
tommivikkula opened this issue Sep 28, 2020 · 8 comments
Closed

AZ ML CLI login issue with MSI #2461

tommivikkula opened this issue Sep 28, 2020 · 8 comments
Labels
extension/ml Machine Learning ML-MLOps AreaPath Service Attention This issue is responsible by Azure service team.

Comments

@tommivikkula
Copy link

tommivikkula commented Sep 28, 2020

  • If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at Azure/azure-cli

Extension name (the extension in question)

azure-cli-ml

Description of issue (in as much detail as possible)

I'm trying to run a azure ml cli command using MSI identity the following way in a Azure VM (during a CICD pipeline run):

az login --identity ...
az ml datastore attach-blob -a ... \
                                  -c ... \
                                  -n ... \
                                  -w ... \
                                  -g ... \
				--storage-account-subscription-id ... \
				--storage-account-resource-group ...

And getting this error:

msrestazure.azure_active_directory : MSI: Token retrieved
cli.azure.cli.core.util : AuthenticationException:
	Message: Could not retrieve user token. Please run 'az login'
	InnerException 'MSIAuthentication' object has no attribute '_token_retriever'
	ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}
AuthenticationException:
	Message: Could not retrieve user token. Please run 'az login'
	InnerException 'MSIAuthentication' object has no attribute '_token_retriever'
	ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}
az_command_data_logger : exit code: 1

I get this similar error in other Azure ML CLI commands like az ml computetarget create amlcompute ...

Versions used:

azure-cli                      2.12.0
azure-cli-ml                   1.14.0

However this was reproduceable at least with azure-cli 2.11.3 and 2.10.1

az login is working fine. This I know because ordinary Azure CLI commands are ran successfully during the same AZDO pipeline run in the same VM. Commands like this:

# This is working fine
az storage container list --auth-mode login ...

I also ran the same az ml datastore attach-blob commands locally using az login with my own user credentials (and same az-cli, az-ml-cli versions). These commands worked fine. This leads me to believe this issue concerns az ml cli and when using msi authentication


@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 28, 2020
@yonzhan yonzhan added extension/ml Machine Learning Service Attention This issue is responsible by Azure service team. and removed needs-triage This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 28, 2020
@ghost
Copy link

ghost commented Sep 28, 2020

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

@yonzhan
Copy link
Collaborator

yonzhan commented Sep 28, 2020

ml

@v-strudm-msft v-strudm-msft added the ML-MLOps AreaPath label Oct 26, 2020
@jiasli
Copy link
Member

jiasli commented Nov 10, 2020

I installed azure-cli-ml with

az extension add -n azure-cli-ml

Judging by the error 'MSIAuthentication' object has no attribute '_token_retriever', I searched for the usage of _token_retriever under ~\.azure\cliextensions\azure-cli-ml. There are 2 occurrences which may cause this issue:

  • azureml\core\authentication.py:1820
  • azureml\core\authentication.py:1869
auth, _, _ = profile_object.get_login_credentials(resource)
access_token = auth._token_retriever()[1]

This is exactly the same coding fault as Azure/azure-cli#15805 (comment).

@azureml-github, please refer to my comment to retrieve the token in the correct way.

@hyssh
Copy link

hyssh commented Jan 14, 2021

Hello, I have been used Azure-ml-cli since 1.18.0 and faced this issue for long time. Do we know when this will be fixed and released?

@singankit
Copy link
Contributor

@hyssh Thanks for the query. We are working on the fix and should be available two weeks from now with the next azure-cli-ml release.

@elnica6-zz
Copy link

Is there a particular release this fixed is slated to go out with?

@singankit
Copy link
Contributor

This has been fixed in azure-cli-ml extension version 1.21.0. Please update the extension using the below command:

az extension update --name azure-cli-ml

@v-strudm-msft
Copy link

Per the above posts, this issue was resolved with a fix at least two months ago. We'll now close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extension/ml Machine Learning ML-MLOps AreaPath Service Attention This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

7 participants