Support for v2 of awscli

[hajapy]

A discussion was started about supporting v2 of the awscli: https://gitter.im/conda-forge/conda-forge.github.io?at=60bfbd6a9cc69444a029b1dc.

There are a few challenges I will highlight here:

1. A pre-requisite for awscli v2.2.x (but not v2.1.x or v2.0.x) is awscrt. An effort was started on creating a recipe for this here, but it has stalled due to a gcc compiler issue: add awscrt staged-recipes#14664
2. Another pre-requisite is prompt-toolkit >=2.0.0,<3.0.0, which isn't built in conda forge (3.0.4+ is built)
3. v1 and v2 of the awscli are both receiving updates, so while that is the case we may need to support both (perhaps via two feedstocks?). There is a confusingly named feedstock https://github.com/conda-forge/awscliv2-feedstock that isn't simply awscli 2.x but rather a different wrapper application (not provided by aws).
4. As of v2, aws have stopped publishing awscli to pypi, for reasons mentioned here: publish v2 to PyPI aws/aws-cli#4947, so the recipe will need tweaks to use github archives, as well as changes to how the fixed offset from botocore is computed.

Options:

a. Drop support for awscli v1 in cf in favor of v2, but need to resolve issues 1 & 2.
b. Continue to support both versions of the awscli -> most likely move to multiple feedstocks (what should the second one be called in light of the pre-existing but different feedstock?)
c. ??

[jakirkham]

cc @mariusvniekerk (as this seems like something you might be interested in 🙂)

[manics]

Great to see this! I've got a hacked fork of this and the botocore repo for v2:

• https://github.com/manics/awscli-feedstock/tree/v2 (https://anaconda.org/manics/awscli/files)
• https://github.com/manics/botocore-feedstock/tree/v2 (https://anaconda.org/manics/botocore/files)

Luckily I happened to build those packages before the awscrt dependency was introduced. In addition to the issues already mentioned:

• botocore2 doesn't have any tags, so you need to pin by git commit hash and examine the version field in botocore/__init__.py to find a match for the AWSCLIv2 requirement
• Autocomplete in awscli v2 requires running a command to generate the completions after awscli-v2 is installed. You can do it at packaging time (thanks to the homebrew package for figuring that out as it's poorly documented), but the downside is you need all the run-time packages to be installed in build dependencies too.
• Some of the dependency versions specified by awscli weren't available, but it seems to work if I change them to something present in conda-forge.

I'm happy to help out with this.

[hajapy]

@manics thanks for that info! Didn't notice that botocore dependency from the v2 branch. Those versions are not even tagged :/

I'm curious how are boto3, botocore and awscli supposed to co-exist in an environment if awscli depends on botocore 2.0.0devX and boto3 is range constrained to botocore>=1.20.90,<1.21.0?

Also, I just saw this post aws/aws-cli#6186 where @shaunhickson writes:

2. Provide a source distribution of CLI v2. We are aware that the community wants the CLI available on platforms and through channels beyond those which the team currently supports. By providing a supported source distribution, we intend to enable the community to effectively build and install the CLI on any platform where it’s needed. We will also work with the maintainers of large and popular distribution channels to ensure that the CLI is available via those channels. We will be releasing a draft plan for providing this source distribution soon, and welcome community feedback on it.

Could be relevant to this effort.

[hajapy]

boto/boto3#2571 (comment)

Seems boto3 and awscli v2 should not go into the same virtual environment :/

[mariusvniekerk]

i guess the only way to get them to coexist is to rather drastically alter the package so that awscli v2 vendors botocore v2 in a relocated place. That feels awful.

[tkelman]

Totally willing to entertain work on this as a separate branch. Getting the conda-forge automation to see and respond to upstream v2 releases doesn't seem straightforward right now. If they're working on source distribution of v2, who knows when that will be ready to make use of.

[manics]

New proposal from AWS (via aws/aws-cli#6355)):
https://github.com/kyleknap/aws-cli/blob/source-proposal/proposals/source-install.md

Official way to build/install:

./configure
make
make install

Alternative PEP 517 way to build/install:

pip wheel . --wheel-dir dist/
pip install dist/awscli-2.2.1-py3-none-any.whl

botocore and s3transfer will be vendored

[mariusvniekerk]

That seems promising. Does that install to the same import place as before?

[manics]

I've built a conda package using https://github.com/kyleknap/aws-cli/tree/poc-sdist as linked from aws/aws-cli#6352

To test it: conda install -c manics/label/dev -c conda-forge awscli=2

botocore and s3transfer are installed as Python submodules awscli.botocore awscli.s3transfer. The awscli imports are the same, test.imports in the recipe is unchanged.

This is the diff from my current v2 package: manics/awscli-feedstock@v2...manics:proposal-6352

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 09ba19d..0375ebd 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -1,15 +1,15 @@
 {% set name = "awscli" %}
-{% set version = "2.1.38" %}
-
-{% set botocore_pin = "2.0.0dev106" %}
+{% set version = "2.proposal.6352" %}
+{% set git_rev = "391e55079f2259bfb8bb06525720e3661ca066ea" %}
 
 package:
   name: {{ name|lower }}
   version: {{ version }}
 
 source:
-  git_url: https://github.com/aws/aws-cli.git
-  git_rev: {{ version }}
+  git_url: https://github.com/kyleknap/aws-cli.git
+  # https://github.com/kyleknap/aws-cli/tree/poc-sdist
+  git_rev: {{ git_rev }}
 
 build:
   number: 0
@@ -17,7 +17,6 @@ build:
   # https://github.com/amancevice/homebrew-core/blob/78db076bcc232c93d3df78209e72e1d6b9a358b1/Formula/awscli.rb#L47
   script: |
     {{ PYTHON }} -m pip install . --no-deps -vv
-    {{ PYTHON }} scripts/gen-ac-index --include-builtin-index --index-location /tmp/ignore.ac
 
 requirements:
   host:
@@ -28,37 +27,47 @@ requirements:
     # dependencies too. YAML anchors don't support merging lists so we have to
     # duplicate everything :-(
     # Commented lines are the original requirements from
-    # https://github.com/aws/aws-cli/blob/2.1.38/setup.py#L27-L37
-    - botocore =={{ botocore_pin }}
+    # https://github.com/kyleknap/aws-cli/blob/391e55079f2259bfb8bb06525720e3661ca066ea/setup.cfg#L29-L39
     # - colorama >=0.2.5,<0.4.4
     - colorama >=0.2.5,<0.5.0
     # - docutils >=0.10,<0.16
     - docutils >=0.10,<0.18
     # - cryptography >=3.3.2,<3.4.0
     - cryptography >=3.3.2,<3.5.0
-    - s3transfer >=0.3.0,<0.4.0
     # - ruamel.yaml >=0.15.0,<0.16.0
     - ruamel.yaml >=0.15.0,<0.17.0
     - wcwidth <0.2.0
     - prompt_toolkit >=2.0.0,<3.0.0
     - distro >=1.5.0,<1.6.0
+    # - awscrt==0.11.13
+    - awscrt >=0.11.13,<0.12.0
+    - python-dateutil>=2.1,<3.0.0
+    - jmespath>=0.7.1,<1.0.0
+    - urllib3>=1.25.4,<1.27
+
+  build:
+    - setuptools >=40.6.0
+    - wheel
 
   run:
     # Commented lines are the original requirements from
-    # https://github.com/aws/aws-cli/blob/2.1.38/setup.py#L27-L37
-    - botocore =={{ botocore_pin }}
+    # https://github.com/kyleknap/aws-cli/blob/391e55079f2259bfb8bb06525720e3661ca066ea/setup.cfg#L29-L39
     # - colorama >=0.2.5,<0.4.4
     - colorama >=0.2.5,<0.5.0
     # - docutils >=0.10,<0.16
     - docutils >=0.10,<0.18
     # - cryptography >=3.3.2,<3.4.0
     - cryptography >=3.3.2,<3.5.0
-    - s3transfer >=0.3.0,<0.4.0
     # - ruamel.yaml >=0.15.0,<0.16.0
     - ruamel.yaml >=0.15.0,<0.17.0
     - wcwidth <0.2.0
     - prompt_toolkit >=2.0.0,<3.0.0
     - distro >=1.5.0,<1.6.0
+    # - awscrt==0.11.13
+    - awscrt >=0.11.13,<0.12.0
+    - python-dateutil>=2.1,<3.0.0
+    - jmespath>=0.7.1,<1.0.0
+    - urllib3>=1.25.4,<1.27
 
 test:
   commands:

This is the diff from this repo (v1): master...manics:proposal-6352

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 0ce040f..0375ebd 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -1,57 +1,110 @@
 {% set name = "awscli" %}
-{% set version = "1.20.34" %}
-{% set hash = "8d6dd12d84e5d9e071e25389776d5539a642dfe232caefacfcdb649ba4234fd6" %}
-
-{% set vmajor,vminor,vpatch = version.split('.') %}
-# ALWAYS DOUBLE-CHECK THESE OFFSETS AGAINST UPSTREAM setup.py
-{% set botocore_major = vmajor | int + 0 %}
-{% set botocore_minor = vminor | int + 1 %}
-{% set botocore_patch = vpatch | int + 0 %}
-{% set botocore_pin = botocore_major ~ '.' ~ botocore_minor ~ '.' ~ botocore_patch %}
+{% set version = "2.proposal.6352" %}
+{% set git_rev = "391e55079f2259bfb8bb06525720e3661ca066ea" %}
 
 package:
   name: {{ name|lower }}
   version: {{ version }}
 
 source:
-  fn: {{ name }}-{{ version }}.tar.gz
-  url: https://pypi.org/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
-  sha256: {{ hash }}
+  git_url: https://github.com/kyleknap/aws-cli.git
+  # https://github.com/kyleknap/aws-cli/tree/poc-sdist
+  git_rev: {{ git_rev }}
 
 build:
   number: 0
-  script: {{ PYTHON }} -m pip install . --no-deps -vv
+  noarch: python
+  # https://github.com/amancevice/homebrew-core/blob/78db076bcc232c93d3df78209e72e1d6b9a358b1/Formula/awscli.rb#L47
+  script: |
+    {{ PYTHON }} -m pip install . --no-deps -vv
 
 requirements:
   host:
-    - python
-#    - python >=3.6
+    - python >=3.6
     - pip
+    # The build process generates the autocomplete index (scripts/gen-ac-index)
+    # which requires importing awscli modules, so we need to install all runtime
+    # dependencies too. YAML anchors don't support merging lists so we have to
+    # duplicate everything :-(
+    # Commented lines are the original requirements from
+    # https://github.com/kyleknap/aws-cli/blob/391e55079f2259bfb8bb06525720e3661ca066ea/setup.cfg#L29-L39
+    # - colorama >=0.2.5,<0.4.4
+    - colorama >=0.2.5,<0.5.0
+    # - docutils >=0.10,<0.16
+    - docutils >=0.10,<0.18
+    # - cryptography >=3.3.2,<3.4.0
+    - cryptography >=3.3.2,<3.5.0
+    # - ruamel.yaml >=0.15.0,<0.16.0
+    - ruamel.yaml >=0.15.0,<0.17.0
+    - wcwidth <0.2.0
+    - prompt_toolkit >=2.0.0,<3.0.0
+    - distro >=1.5.0,<1.6.0
+    # - awscrt==0.11.13
+    - awscrt >=0.11.13,<0.12.0
+    - python-dateutil>=2.1,<3.0.0
+    - jmespath>=0.7.1,<1.0.0
+    - urllib3>=1.25.4,<1.27
+
+  build:
+    - setuptools >=40.6.0
+    - wheel
 
   run:
-    - python
-#    - python >=3.6
-    - botocore =={{ botocore_pin }}
-    - colorama >=0.2.5,<0.4.4
-    - docutils >=0.10,<0.16
-    - rsa >=3.1.2,<4.8
-    - s3transfer >=0.5.0,<0.6.0
-    - pyyaml >=3.10,<5.5
+    # Commented lines are the original requirements from
+    # https://github.com/kyleknap/aws-cli/blob/391e55079f2259bfb8bb06525720e3661ca066ea/setup.cfg#L29-L39
+    # - colorama >=0.2.5,<0.4.4
+    - colorama >=0.2.5,<0.5.0
+    # - docutils >=0.10,<0.16
+    - docutils >=0.10,<0.18
+    # - cryptography >=3.3.2,<3.4.0
+    - cryptography >=3.3.2,<3.5.0
+    # - ruamel.yaml >=0.15.0,<0.16.0
+    - ruamel.yaml >=0.15.0,<0.17.0
+    - wcwidth <0.2.0
+    - prompt_toolkit >=2.0.0,<3.0.0
+    - distro >=1.5.0,<1.6.0
+    # - awscrt==0.11.13
+    - awscrt >=0.11.13,<0.12.0
+    - python-dateutil>=2.1,<3.0.0
+    - jmespath>=0.7.1,<1.0.0
+    - urllib3>=1.25.4,<1.27
 
 test:
+  commands:
+    - aws_completer
+  # TODO: Check autocomplete index exists:
+  #   lib/python3.x/site-packages/awscli/data/ac.index
   imports:
     - awscli
+    - awscli.autocomplete
+    - awscli.autoprompt
+    - awscli.bcdoc
     - awscli.customizations
+    - awscli.customizations.cloudformation
     - awscli.customizations.cloudtrail
+    - awscli.customizations.codeartifact
     - awscli.customizations.codedeploy
     - awscli.customizations.configservice
     - awscli.customizations.configure
     - awscli.customizations.datapipeline
+    - awscli.customizations.dlm
+    - awscli.customizations.dynamodb
     - awscli.customizations.ec2
+    - awscli.customizations.ecs
+    - awscli.customizations.eks
     - awscli.customizations.emr
+    - awscli.customizations.emrcontainers
     - awscli.customizations.gamelift
+    - awscli.customizations.history
+    - awscli.customizations.lightsail
+    - awscli.customizations.logs
     - awscli.customizations.s3
     - awscli.customizations.s3.syncstrategy
+    - awscli.customizations.servicecatalog
+    - awscli.customizations.sso
+    - awscli.customizations.wizard
+    - awscli.customizations.wizard.ui
+    - awscli.customizations.wizard.wizards
 
 about:
   home: https://aws.amazon.com/cli/
@@ -64,5 +117,4 @@ about:
 
 extra:
   recipe-maintainers:
-    - hajapy
-    - tkelman
+    - manics

[manics]

The proposal was accepted, follow aws/aws-cli#6785 for updates on the implementation

[manics]

At some point in the past year:

• pep517 support was added
• The unreleased versions of botocore and s3transfer were vendored under awscli

This means we should have everything needed to package v2: #1204

[simone-pignotti]

Hi, any news about the timeline of v2 package availability? Thanks!

[tkelman]

There were test failures where #1204 was left off, and I guess the author didn't have the time to resolve them. Looks like the logs have expired and it's a bit stale, so could be taken over by someone else. Is there functionality that's available in v2 but not in v1? Otherwise v1 is still actively supported and has new releases basically daily.

[simone-pignotti]

Yes there are a few new features in v2. Nothing essential as far as I am concerned, but AWS now recommends installing v2 in their docs and in the help messages:

$ aws s3api
Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more information, see the AWS CLI version 2 installation instructions at: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

usage: [...]

I took a look at the current status and unfortunately I don't think I can take this task as of now, sorry.

[manics]

Thanks to @xhochy this was done! It's in a separate feedstock repo https://github.com/conda-forge/awscli2-feedstock but it pushes to https://anaconda.org/conda-forge/awscli so conda install awscli still works to install v2