Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add commit hook tooling. #346

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .bumpversion.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prerelease>a|b|rc)(?P<num>\d+))?
serialize =
serialize =
{major}.{minor}.{patch}{prerelease}{num}
{major}.{minor}.{patch}
commit = False
Expand All @@ -13,7 +13,7 @@ tag = False
[bumpversion:part:prerelease]
first_value = a
optional_value = final
values =
values =
a
b
rc
Expand All @@ -25,4 +25,3 @@ first_value = 1
[bumpversion:file:setup.py]

[bumpversion:file:dbt/adapters/spark/__version__.py]

12 changes: 12 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[flake8]
select =
E
W
F
ignore =
W503 # makes Flake8 work like black
W504
E203 # makes Flake8 work like black
E741
E501
exclude = test
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ updates:
directory: "/"
schedule:
interval: "daily"
rebase-strategy: "disabled"
rebase-strategy: "disabled"
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ assignees: ''

---

### TBD
### TBD
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ resolves #
- [ ] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-spark next" section.
- [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-spark next" section.
2 changes: 1 addition & 1 deletion .github/workflows/jira-creation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ name: Jira Issue Creation
on:
issues:
types: [opened, labeled]

permissions:
issues: write

Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/jira-label.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ name: Jira Label Mirroring
on:
issues:
types: [labeled, unlabeled]

permissions:
issues: read

Expand All @@ -24,4 +24,3 @@ jobs:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}

2 changes: 1 addition & 1 deletion .github/workflows/jira-transition.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ jobs:
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
26 changes: 9 additions & 17 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,19 +37,10 @@ defaults:

jobs:
code-quality:
name: ${{ matrix.toxenv }}
name: code-quality

runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
toxenv: [flake8]

env:
TOXENV: ${{ matrix.toxenv }}
PYTEST_ADDOPTS: "-v --color=yes"

steps:
- name: Check out the repository
uses: actions/checkout@v2
Expand All @@ -58,18 +49,19 @@ jobs:

- name: Set up Python
uses: actions/setup-python@v2
with:
with:
python-version: '3.8'

- name: Install python dependencies
run: |
sudo apt-get install libsasl2-dev
pip install --user --upgrade pip
pip install tox
pip --version
tox --version
- name: Run tox
run: tox
pip install -r dev_requirements.txt
pre-commit --version
mypy --version
dbt --version
- name: pre-commit hooks
run: pre-commit run --all-files --show-diff-on-failure

unit:
name: unit test / python ${{ matrix.python-version }}
Expand Down Expand Up @@ -153,7 +145,7 @@ jobs:
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008

- name: Check if this is an alpha version
id: check-is-alpha
run: |
Expand Down
27 changes: 13 additions & 14 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,23 @@ name: Build and Release

on:
workflow_dispatch:

# Release version number that must be updated for each release
env:
version_number: '0.20.0rc2'

jobs:
jobs:
Test:
runs-on: ubuntu-latest
steps:
- name: Setup Python
uses: actions/[email protected]
with:
with:
python-version: '3.8'

- uses: actions/checkout@v2

- name: Test release
- name: Test release
run: |
python3 -m venv env
source env/bin/activate
Expand All @@ -38,9 +38,9 @@ jobs:
steps:
- name: Setup Python
uses: actions/[email protected]
with:
with:
python-version: '3.8'

- uses: actions/checkout@v2

- name: Bumping version
Expand All @@ -60,7 +60,7 @@ jobs:
author_email: '[email protected]'
message: 'Bumping version to ${{env.version_number}}'
tag: v${{env.version_number}}

# Need to set an output variable because env variables can't be taken as input
# This is needed for the next step with releasing to GitHub
- name: Find release type
Expand All @@ -69,7 +69,7 @@ jobs:
IS_PRERELEASE: ${{ contains(env.version_number, 'rc') || contains(env.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE

- name: Create GitHub release
uses: actions/create-release@v1
env:
Expand All @@ -88,7 +88,7 @@ jobs:
# or
$ pip install "dbt-spark[PyHive]==${{env.version_number}}"
```

PypiRelease:
name: Pypi release
runs-on: ubuntu-latest
Expand All @@ -97,13 +97,13 @@ jobs:
steps:
- name: Setup Python
uses: actions/[email protected]
with:
with:
python-version: '3.8'

- uses: actions/checkout@v2
with:
ref: v${{env.version_number}}

- name: Release to pypi
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
Expand All @@ -116,4 +116,3 @@ jobs:
pip install twine wheel setuptools
python setup.py sdist bdist_wheel
twine upload --non-interactive dist/dbt_spark-${{env.version_number}}-py3-none-any.whl dist/dbt-spark-${{env.version_number}}.tar.gz

2 changes: 0 additions & 2 deletions .github/workflows/stale.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,3 @@ jobs:
stale-pr-message: "This PR has been marked as Stale because it has been open for 180 days with no activity. If you would like the PR to remain open, please remove the stale label or comment on the PR, or it will be closed in 7 days."
# mark issues/PRs stale when they haven't seen activity in 180 days
days-before-stale: 180
# ignore checking issues with the following labels
exempt-issue-labels: "epic, discussion"
20 changes: 10 additions & 10 deletions .github/workflows/version-bump.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# **what?**
# This workflow will take a version number and a dry run flag. With that
# it will run versionbump to update the version number everywhere in the
# it will run versionbump to update the version number everywhere in the
# code base and then generate an update Docker requirements file. If this
# is a dry run, a draft PR will open with the changes. If this isn't a dry
# run, the changes will be committed to the branch this is run on.

# **why?**
# This is to aid in releasing dbt and making sure we have updated
# This is to aid in releasing dbt and making sure we have updated
# the versions and Docker requirements in all places.

# **when?**
# This is triggered either manually OR
# This is triggered either manually OR
# from the repository_dispatch event "version-bump" which is sent from
# the dbt-release repo Action

Expand All @@ -25,11 +25,11 @@ on:
is_dry_run:
description: 'Creates a draft PR to allow testing instead of committing to a branch'
required: true
default: 'true'
default: 'true'
repository_dispatch:
types: [version-bump]

jobs:
jobs:
bump:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -58,19 +58,19 @@ jobs:
sudo apt-get install libsasl2-dev
python3 -m venv env
source env/bin/activate
pip install --upgrade pip
pip install --upgrade pip

- name: Create PR branch
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
run: |
git checkout -b bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git push origin bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git branch --set-upstream-to=origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID

- name: Bumping version
run: |
source env/bin/activate
pip install -r dev_requirements.txt
pip install -r dev_requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{steps.variables.outputs.VERSION_NUMBER}} major
git status

Expand Down Expand Up @@ -100,4 +100,4 @@ jobs:
draft: true
base: ${{github.ref}}
title: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
53 changes: 41 additions & 12 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,18 +1,47 @@
.hive-metastore/
.spark-warehouse/
*.egg-info
env/
*.pyc
# Byte-compiled / optimized / DLL files
__pycache__
*.py[cod]
*$py.class

# Distribution / packaging
.Python
build/
env*/
dbt_env/
dist/
*.egg-info
logs/


# Unit test
.tox/
.env
test.env


# Django stuff
*.log

# Mypy
*.pytest_cache/

# Vim
*.sw*

# Pyenv
.python-version

# pycharm
.idea/
build/
dist/
dbt-integration-tests
test/integration/.user.yml

# MacOS
.DS_Store
test.env

# vscode
.vscode
*.log
logs/

# other
.hive-metastore/
.spark-warehouse/
dbt-integration-tests
test/integration/.user.yml
Loading