azureml-pipeline incompatible with azure-cli

[tonyskidmore]

• Package Name:
  azureml-pipeline
• Package Version:
  azureml-pipeline==1.47.0
• Operating System:
  Linux
• Python Version:
  3.8.10

Describe the bug
Unable to install azureml-pipeline package alongside azure-cli==2.42.0, azureml-core==1.47.0 and azureml-mlflow==1.47.0. It could be that there are subsequent errors trying to install other updated azureml-pipeline* packages but this throws the initial error.

To Reproduce
Steps to reproduce the behavior:

python3 -m venv ~/venvs/azureml-pipeline
source ~/venvs/azureml-pipeline/bin/activate
pip install pip --upgrade
cat requirements.txt
azure-cli==2.42.0
azureml-core==1.47.0
azureml-mlflow==1.47.0
pip install -r requirements.txt
# after installation try to install azureml-pipeline
pip install azureml-pipeline==1.47.0

Expected behavior
Package should be installed without error.

Screenshots

  Attempting uninstall: msal-extensions
    Found existing installation: msal-extensions 1.0.0
    Uninstalling msal-extensions-1.0.0:
      Successfully uninstalled msal-extensions-1.0.0
  Attempting uninstall: azure-identity
    Found existing installation: azure-identity 1.11.0
    Uninstalling azure-identity-1.11.0:
      Successfully uninstalled azure-identity-1.11.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
azure-cli-core 2.42.0 requires msal-extensions~=1.0.0, but you have msal-extensions 0.3.1 which is incompatible.

Additional context
We have been using this setup for some time in our Azure DevOps pipeline. We had v2.35.0 of the azure-cli package installed due to some previously encountered dependency issue but recently our pipeline started failing due to a Mend vulnerability scanning for CVE-2022-39327. I have tried various combinations of the libraries and have not been able to bring the azure-cli and azureml-* packages up to date.

A snippet of the relevant requirements of our code with the required libraries in use are as follows:

import azureml.core
from azureml.core import Dataset, Environment, Workspace, Datastore, ScriptRunConfig
from azureml.core.authentication import AzureCliAuthentication
from azureml.data.datapath import DataPath
from azure.storage.blob import BlobServiceClient
from azureml.core.model import Model
from azureml.core.compute import ComputeTarget
from azureml.core.compute_target import ComputeTargetException

from azureml.pipeline.core import Pipeline
from azureml.pipeline.steps import PythonScriptStep

cli_auth = AzureCliAuthentication()

If I try using azure-cli at the system level rather than a package, as suggested in the similar issue #24787 then I get the error:

{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "azure-cli package is not installed. AzureCliAuthentication requires azure-cli>=2.30.0 to be installed in the same python environment where azureml-sdk is installed."
    }
}

This may have some relation to #23927.

[xiangyan99]

@azureml-github

[brynn-code]

I also met this issue, and find the root cause is the dependencies line below:
azureml-pipeline-steps -> azureml-train-automl-client -> azureml-dataset-runtime -> azureml-dataprep -> azure-identity==1.7.0 -> msal-extensions~=0.3.0
azure-cli-core==2.42.0 -> msal-extensions~=1.0.0

The azure-identity==1.11.0 required msal-extensions<2.0.0,>=0.3.0 seems could fix this issue, however, azureml-dataprep has pinned azure-identity must be 1.7.0 which cause the conflict.

[luigiw]

@wangchao1230 can you help to triage this issue? Thank you.

[davidblom603]

I have the same issue, poetry is not able to resolve the version conflict:

$ poetry add azure-cli azureml-sdk 
Using version ^2.44.1 for azure-cli
Using version ^1.48.0 for azureml-sdk

Updating dependencies
Resolving dependencies... (4.1s)

Because no versions of azureml-sdk match >1.48.0,<2.0.0
 and azureml-sdk (1.48.0) depends on azureml-core (>=1.48.0,<1.49.0), azureml-sdk (>=1.48.0,<2.0.0) requires azureml-core (>=1.48.0,<1.49.0).
And because no versions of azureml-core match >1.48.0,<1.49.0
 and azureml-core (1.48.0) depends on azure-mgmt-storage (>=16.0.0,<21.0.0), azureml-sdk (>=1.48.0,<2.0.0) requires azure-mgmt-storage (>=16.0.0,<21.0.0).
And because azure-cli (2.44.1) depends on azure-mgmt-storage (>=21.0.0,<21.1.0)
 and no versions of azure-cli match >2.44.1,<3.0.0, azureml-sdk (>=1.48.0,<2.0.0) is incompatible with azure-cli (>=2.44.1,<3.0.0).
So, because q-trader depends on both azure-cli (^2.44.1) and azureml-sdk (^1.48.0), version solving failed.

[wangchao1230]

Newer version of azureml-dataprep has relaxed the dependency requirement from azure-identity==1.7.0 to azure-identity>=1.7.0.
It should have resolved the issue.

@davidblom603 Please give it a try.

[davidblom603]

@wangchao1230 thanks for your message. poetry still shows this error:

Because no versions of azure-cli match >2.44.1,<3.0.0
 and azure-cli (2.44.1) depends on azure-mgmt-storage (>=21.0.0,<21.1.0), azure-cli (>=2.44.1,<3.0.0) requires azure-mgmt-storage (>=21.0.0,<21.1.0).
Because azureml-core (1.48.0) depends on azure-mgmt-storage (>=16.0.0,<21.0.0)
 and no versions of azureml-core match >1.48.0,<1.49.0, azureml-core (>=1.48.0,<1.49.0) requires azure-mgmt-storage (>=16.0.0,<21.0.0).
Thus, azure-cli (>=2.44.1,<3.0.0) is incompatible with azureml-core (>=1.48.0,<1.49.0).
And because azureml-sdk (1.48.0) depends on azureml-core (>=1.48.0,<1.49.0)
 and no versions of azureml-sdk match >1.48.0,<2.0.0, azure-cli (>=2.44.1,<3.0.0) is incompatible with azureml-sdk (>=1.48.0,<2.0.0).
So, because air-clustering depends on both azureml-sdk (^1.48.0) and azure-cli (^2.44.1), version solving failed.

the problem seems to be that azure-cli requires azure-mgmt-storage (>=21.0.0,<21.1.0), and azureml-core requires azure-mgmt-storage (>=16.0.0,<21.0.0), which is a conflict.

[tonyskidmore]

The following combination now appears to work:

python3 -m venv ~/venvs/azureml-pipeline
source ~/venvs/azureml-pipeline/bin/activate
pip install pip --upgrade
cat requirements.txt
azure-cli==2.42.0
azureml-core==1.48.0
azureml-mlflow==1.48.0
pip install -r requirements.txt
# after installation try to install azureml-pipeline
pip install azureml-pipeline==1.48.0

However, going above azure-cli==2.42.0 (tested with 2.43.0 and 2.44.1) ends with the following:

ERROR: Cannot install -r requirements.txt (line 1) and -r requirements.txt (line 2) because these package versions have conflicting dependencies.

The conflict is caused by:
    azure-cli 2.43.0 depends on azure-mgmt-storage~=21.0.0
    azureml-core 1.48.0 depends on azure-mgmt-storage<21.0.0 and >=16.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @simorenoh @gahl-levy @bambriz @azureml-github @Azure/azure-ml-sdk.

[adrshm91]

I have the same problem. I got it working with

azure-cli==2.42.0
azureml-core==1.48.0
azureml-defaults==1.48.0
azureml-mlflow==1.48.0
azureml-telemetry==1.48.0

It would be nice to have the fix for latest versions.

[jaga-work]

Hi @tonyskidmore, is this issue is resolved. if not, could you please share the latest snippets you are using to reproduce the issue. Thank you.