Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: unify dependency management + hatch scripts #7079

Merged
merged 8 commits into from
Feb 26, 2024
Merged

ci: unify dependency management + hatch scripts #7079

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5399ed9
ci: unify dependency management + hatch scripts
wochinge Feb 23, 2024
0d4dc4e
ci: migrate readme sync
wochinge Feb 23, 2024
328f807
build: migrate snippets
wochinge Feb 23, 2024
cfce201
ci: pin hatch
wochinge Feb 23, 2024
86ce71d
ci: make Python version more explicit + quote
wochinge Feb 23, 2024
6363a4b
Merge branch 'main' into ci/full-hatch-setup
wochinge Feb 26, 2024
09fb25a
Merge branch 'main' into ci/full-hatch-setup
silvanocerza Feb 26, 2024
313fa45
ci: add scripts with parameters to hatch
wochinge Feb 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/docstring_labeler.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ on:
paths:
- "haystack/**/*.py"

env:
PYTHON_VERSION: "3.11"

jobs:
label:
runs-on: ubuntu-latest
Expand All @@ -24,7 +27,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "${{ env.PYTHON_VERSION }}"

- name: Get docstrings
id: base-docstrings
Expand Down
14 changes: 5 additions & 9 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ on:
env:
PYTHON_VERSION: "3.8"
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
HATCH_VERSION: "1.9.3"

jobs:
run:
Expand All @@ -28,15 +29,10 @@ jobs:

- uses: actions/setup-python@v5
with:
wochinge marked this conversation as resolved.
Show resolved Hide resolved
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added quotes as this otherwise leads to issues with >= 3.10


- name: Install dependencies
run: |
sudo apt-get update
sudo apt install ffmpeg # for local Whisper tests

- name: Install Haystack and the dependencies needed for tests
run: pip install -r test/test_requirements.txt
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Run tests
run: pytest e2e
run: hatch run test:e2e
silvanocerza marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 2 additions & 1 deletion .github/workflows/license_compliance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ on:

env:
CORE_DATADOG_API_KEY: ${{ secrets.CORE_DATADOG_API_KEY }}
PYTHON_VERSION: "3.10"

jobs:
license_check_direct:
Expand All @@ -25,7 +26,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "${{ env.PYTHON_VERSION }}"

- name: Get direct dependencies
run: |
Expand Down
21 changes: 10 additions & 11 deletions .github/workflows/linting.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on:

env:
PYTHON_VERSION: "3.8"
HATCH_VERSION: "1.9.3"

jobs:
mypy:
Expand All @@ -35,19 +36,17 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install Haystack
run: pip install .[dev]

- name: Install the dependencies needed for tests
run: pip install -r test/test_requirements.txt
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Mypy
if: steps.files.outputs.any_changed == 'true'
run: |
mkdir .mypy_cache
mypy --install-types --non-interactive --cache-dir=.mypy_cache/ ${{ steps.files.outputs.all_changed_files }}
# can't use hatch script due to positional argument
hatch run test:mypy --install-types --non-interactive --cache-dir=.mypy_cache/ ${{ steps.files.outputs.all_changed_files }}

pylint:
runs-on: ubuntu-latest
Expand All @@ -68,12 +67,12 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install Haystack and the dependencies needed for tests
run: pip install -r test/test_requirements.txt
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Pylint
if: steps.files.outputs.any_changed == 'true'
run: |
pylint -ry -j 0 ${{ steps.files.outputs.all_changed_files }}
hatch run test:pylint -ry -j 0 ${{ steps.files.outputs.all_changed_files }}
wochinge marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion .github/workflows/minor_version_release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install release_docs.py dependencies
run: pip install requests
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/pypi_release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ on:
tags:
- "v[0-9].[0-9]+.[0-9]+*"

env:
HATCH_VERSION: "1.9.3"

jobs:
release-on-pypi:
runs-on: ubuntu-latest
Expand All @@ -14,7 +17,7 @@ jobs:
uses: actions/checkout@v4

- name: Install Hatch
run: pip install hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Build Haystack
run: hatch build
Expand Down
14 changes: 8 additions & 6 deletions .github/workflows/readme_sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ on:
branches:
- main

env:
HATCH_VERSION: "1.9.3"
PYTHON_VERSION: "3.10"

jobs:
sync:
runs-on: ubuntu-latest
Expand All @@ -18,20 +22,18 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -U haystack-pydoc-tools
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Generate API docs
env:
# This is necessary to fetch the documentation categories
# from Readme.io as we need them to associate the slug
# in config files with their id.
README_API_KEY: ${{ secrets.README_API_KEY }}
run: ./.github/utils/pydoc-markdown.sh
run: hatch run readme:sync

- name: Sync docs with 2.0
# Sync the docs to the `2.0` version on Readme
Expand Down
14 changes: 6 additions & 8 deletions .github/workflows/snippets_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ on:
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
PYTHON_VERSION: "3.8"
HATCH_VERSION: "1.9.3"

jobs:
black:
Expand All @@ -26,7 +27,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install Black
run: |
Expand Down Expand Up @@ -100,13 +101,10 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install snippets dependencies
run: |
pip install --upgrade pip
pip install "." torch
pip install pydantic
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Get changed files
id: files
Expand All @@ -119,7 +117,7 @@ jobs:
run: |
CHANGED_FILES=${{ steps.files.outputs.all_changed_files }}
for file in $CHANGED_FILES; do
python "$file"
hatch run snippets:python "$file"
done

- name: Calculate alert data
Expand Down
41 changes: 20 additions & 21 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ env:
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PYTHON_VERSION: "3.8"
HATCH_VERSION: "1.9.3"

jobs:
black:
Expand All @@ -38,19 +39,13 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install Black
run: |
pip install --upgrade pip
pip install .[dev]
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Check status
run: |
if ! black . --check; then
git status
exit 1
fi
run: hatch run default:format-check

- name: Calculate alert data
id: calculator
Expand Down Expand Up @@ -96,10 +91,14 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install Haystack and the dependencies needed for tests
run: pip install -r test/test_requirements.txt
- name: Install Hatch
run: pip install hatch==${{ env.HATCH_VERSION }}

- name: Install dependencies
# To actually install and sync the dependencies
run: hatch run test:pip list

- uses: actions/cache@v4
with:
Expand All @@ -122,7 +121,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Restore Python dependencies
uses: actions/cache/restore@v4
Expand All @@ -131,7 +130,7 @@ jobs:
key: pip-${{ runner.os }}-${{ github.run_id }}-${{ hashFiles('pyproject.toml', 'test/test_requirements.txt') }}

- name: Run
run: pytest --cov-report xml:coverage.xml --cov="haystack" -m "not integration" test
run: hatch run test:unit

- name: Coveralls
# We upload only coverage for ubuntu as handling both os
Expand Down Expand Up @@ -186,7 +185,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install dependencies
run: |
Expand All @@ -200,7 +199,7 @@ jobs:
key: pip-${{ runner.os }}-${{ github.run_id }}-${{ hashFiles('pyproject.toml', 'test/test_requirements.txt') }}

- name: Run
run: pytest --maxfail=5 -m "integration" test
run: hatch run test:integration

- name: Calculate alert data
id: calculator
Expand Down Expand Up @@ -245,7 +244,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Install dependencies
run: |
Expand All @@ -258,7 +257,7 @@ jobs:
key: pip-${{ runner.os }}-${{ github.run_id }}-${{ hashFiles('pyproject.toml', 'test/test_requirements.txt') }}

- name: Run
run: pytest --maxfail=5 -m "integration" test -k 'not tika'
run: hatch run test:integration-mac

- name: Calculate alert data
id: calculator
Expand Down Expand Up @@ -300,7 +299,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "${{ env.PYTHON_VERSION }}"

- name: Restore Python dependencies
uses: actions/cache/restore@v4
Expand All @@ -309,7 +308,7 @@ jobs:
key: pip-${{ runner.os }}-${{ github.run_id }}-${{ hashFiles('pyproject.toml', 'test/test_requirements.txt') }}

- name: Run
run: pytest --maxfail=5 -m "integration" test -k 'not tika'
run: hatch run test:integration-windows

- name: Calculate alert data
id: calculator
Expand Down
Loading
Loading