Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Python 3.11 #639

Merged
merged 9 commits into from
Oct 26, 2022
Merged

Add support for Python 3.11 #639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4d7337f
Update Dockerfile version, pytest 3.11
parasj Oct 25, 2022
eeb9cfc
Enable support for 3.11 in pyproject.toml
parasj Oct 25, 2022
9677683
Update installation instructions to partition dev dependencies from t…
parasj Oct 25, 2022
6787f98
Remove awscrt dependency
parasj Oct 25, 2022
7c066e1
Add glibc
parasj Oct 25, 2022
937359b
pytype 3.10
parasj Oct 25, 2022
a4f981a
Reduce scope of Python version to <3.12
parasj Oct 25, 2022
13f777a
Install development dependencies in workflows
parasj Oct 26, 2022
44ca728
Add myst-parser
parasj Oct 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,9 @@ jobs:
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- id: 'auth'
uses: 'google-github-actions/auth@v0'
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/poetry-publish-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ jobs:
poetry config repositories.test-pypi https://test.pypi.org/legacy/
poetry config pypi-token.test-pypi ${{ secrets.TEST_PYPI_API_TOKEN }}
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Build package
run: |
Expand Down Expand Up @@ -89,7 +91,9 @@ jobs:
poetry config virtualenvs.path ~/.virtualenvs
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Build package under skyplane-dev name
run: |
Expand Down
14 changes: 11 additions & 3 deletions .github/workflows/poetry-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,11 @@ jobs:
poetry config repositories.test-pypi https://test.pypi.org/legacy/
poetry config pypi-token.test-pypi ${{ secrets.TEST_PYPI_API_TOKEN }}
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Install Dependencies
if: steps.cache.outputs.cache-hit != 'true'
- name: Build package
run: |
Expand Down Expand Up @@ -67,7 +71,9 @@ jobs:
poetry config virtualenvs.path ~/.virtualenvs
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Build package under skyplane-dev name
run: |
Expand Down Expand Up @@ -105,7 +111,9 @@ jobs:
poetry config virtualenvs.path ~/.virtualenvs
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Build package under skyplane name
run: |
Expand Down
22 changes: 16 additions & 6 deletions .github/workflows/pytest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ jobs:
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: black
run: poetry run black . --check --line-length=140
Expand All @@ -39,7 +41,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v1
- name: Install poetry
Expand All @@ -54,7 +56,9 @@ jobs:
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Run unit tests
run: |
Expand All @@ -78,7 +82,9 @@ jobs:
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Run cloud tests
env:
Expand Down Expand Up @@ -110,7 +116,9 @@ jobs:
with:
creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- name: Run cloud tests
run: |
Expand All @@ -137,7 +145,9 @@ jobs:
poetry config virtualenvs.in-project false
poetry config virtualenvs.path ~/.virtualenvs
- name: Install Dependencies
run: poetry install -E gateway -E solver -E aws -E azure -E gcp
run: |
poetry install -E gateway -E solver -E aws -E azure -E gcp
poetry run pip install -r requirements-dev.txt
if: steps.cache.outputs.cache-hit != 'true'
- id: 'auth'
uses: 'google-github-actions/auth@v0'
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Before you submit a pull request, make sure to complete the following steps:

.. code-block:: console

$ pip install black pytype autoflake
$ pip install -r requirements-dev.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about:

poetry install --only dev

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an issue that Poetry requires Python version compliance with all dev dependencies and will not allow specification of a different Python version.

There's an in progress PR to add support for this, but it won't be production-ready for a while: python-poetry/poetry#5740

Several dev dependencies have not been updated for Python 3.11, most notably pytype which requires python >=3.7.1<3.11.

Until that PR lands, an alternative solution is to extract development CLI dependencies into another file. Thoughts?

$ black -l 140 .
$ pytype --config .pytype.cfg skyplane
$ autoflake --in-place --remove-all-unused-imports --remove-unused-variables --recursive skyplane
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# syntax=docker/dockerfile:1
FROM python:3.10-slim
FROM python:3.11-slim

# install apt packages
RUN --mount=type=cache,target=/var/cache/apt apt update \
&& apt-get install --no-install-recommends -y curl ca-certificates stunnel4 gcc \
&& apt-get install --no-install-recommends -y curl ca-certificates stunnel4 gcc libc-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ Skyplane currently supports the following source and destination endpoints (any
| Azure Blob Storage | :white_check_mark: | :white_check_mark: |
| Local Disk | :white_check_mark: | (in progress) |

Skyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. Please file an issue or ask the contributors via [the #help channel on our Slack](https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q) if you encounter bugs.

# Resources
- [Quickstart](#quickstart)
- [Contributing](https://skyplane.org/en/latest/contributing.html)
Expand Down
6 changes: 6 additions & 0 deletions docs/build_from_source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Building from source

## Prerequisite: install development dependencies
For Skyplane development, we use a variety of tools to check for code quality and to build documentation. To install these tools, run the following command:
```bash
pip install -r requirements-dev.txt
```

## Setting up a developer environment

Skyplane is composed of the client (runs locally on a user's laptop) and gateway VMs (runs in respective clouds). Normally, the gateways use a pre-built nightly Docker image containing the latest build of the Skyplane gateway code (`public.ecr.aws/s6m1p0n8/skyplane:edge`). However, if you modify the gateway source (under `skyplane/gateway`), you will need to rebuild the gateway Docker image and push it to a container registry.
Expand Down
2 changes: 1 addition & 1 deletion docs/quickstart.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ We're ready to use Skyplane! Let's use `skyplane cp` to copy files from AWS to G
.. code-block:: bash

---> 🎸 Ready to rock and roll! Copy some files:
$ skyplane cp s3://... gs://...
$ skyplane cp -r s3://... gs://...

To transfer only new objects, you can instead use `skyplane sync`:

Expand Down
3 changes: 1 addition & 2 deletions docs/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
docutils>=0.18
furo
jupyter-sphinx
myst-nb
myst-parser
sphinx-autobuild
sphinx-autodoc-typehints
sphinx-click
Expand Down
15 changes: 4 additions & 11 deletions docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,13 @@ Skyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. P

If you are interested in contributing, please see our [contributor guide](contributing).

## Skyplane 0.1.0 release (current)
* `skyplane cp` and `skyplane sync` CLI commands to move bulk (>10GB) of data point-to-point.
* Support for intra-cloud (two regions in same cloud) and inter-cloud copies for AWS, GCP and Azure clouds.
* Large object transfer support for AWS and GCP.
* Compression to reduce egress costs.
* End-to-end encryption and optional socket TLS encryption.
* Firewalled VM-to-VM VPCs for AWS and GCP.

## Skyplane 0.2.0 release
## Skyplane 0.2.0 release (current)
* Firewalled VM-to-VM VPCs for Azure.
* Large object transfer support for Azure.
* Integration testing suite in CI.
* Improvements to the scalability of transfers with many files (>10000).

## Upcoming
* Improvements to the scalability of transfers with many files (>10000).
* On-prem transfer support (design needed).
* Broadcast support to multiple destinations.
* Broadcast support to multiple destinations.
* Skyplane managed buckets for cross-cloud object storage.
Loading