Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CHANGELOG and update docs #16

Merged
merged 4 commits into from
Sep 12, 2023
Merged

Add CHANGELOG and update docs #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c09efee
Added CHANGELOG
bobozaur Sep 12, 2023
79f56b4
Added more ETL documentation
bobozaur Sep 12, 2023
e396ed8
Bumped alpha version
bobozaur Sep 12, 2023
fa50568
Added custom actions and split testing in multiple CI jobs
bobozaur Sep 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions .github/actions/exa-cluster/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: "Exasol Cluster"
description: "Instantiates an Exasol database cluster"
inputs:
exasol-version:
description: "Version of the Exasol database"
required: true
num-nodes:
description: "Number of nodes to spawn in the cluster"
required: true
outputs:
no-tls-url:
description: "Connection string for the database with TLS disabled"
value: ${{ steps.connection-strings.outputs.no-tls-url }}
tls-url:
description: "Connection string for the database with TLS enabled"
value: ${{ steps.connection-strings.outputs.tls-url }}
runs:
using: "composite"
steps:
- name: Restore Exasol image
id: cache-docker-exasol
uses: actions/cache@v3
with:
path: ~/ci/cache/docker
key: cache-docker-exasol-${{ inputs.exasol-version }}

- name: Store Exasol image if cache miss
if: steps.cache-docker-exasol.outputs.cache-hit != 'true'
run: docker pull exasol/docker-db:${{ inputs.exasol-version }} && mkdir -p ~/ci/cache/docker && docker image save exasol/docker-db:${{ inputs.exasol-version }} --output ~/ci/cache/docker/exasol-${{ inputs.exasol-version }}.tar
shell: bash

- name: Load Exasol image
run: docker image load --input ~/ci/cache/docker/exasol-${{ inputs.exasol-version }}.tar
shell: bash

- name: Set up Python 3.10
uses: actions/setup-python@v3
with:
python-version: "3.10"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pipenv
shell: bash

- uses: actions/checkout@v3
with:
repository: Exasol/docker-db
ref: ${{ inputs.exasol-version }}

- name: Set up Exasol Cluster
run: |
pipenv install -r exadt_requirements.txt
pipenv run ./exadt create-cluster --root ~/sqlx --create-root sqlx
pipenv run ./exadt init-cluster --image exasol/docker-db:${{ inputs.exasol-version }} --license ./license/license.xml --num-nodes ${{ inputs.num-nodes }} --auto-storage sqlx
pipenv run ./exadt start-cluster sqlx
shell: bash

# The exadt tool always creates the Exasol cluster with this subnet
- name: Create connection strings
id: connection-strings
run: |
DATABASE_URL="exa://sys:[email protected]$NUM_NODES:8563"
echo "no-tls-url=$DATABASE_URL?ssl-mode=disabled" >> $GITHUB_OUTPUT
echo "tls-url=$DATABASE_URL?ssl-mode=required" >> $GITHUB_OUTPUT
shell: bash

# Checkout the main repository again
- uses: actions/checkout@v3
29 changes: 29 additions & 0 deletions .github/actions/free-space/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: "Free disk space"
description: "Deletes language runtimes and implicit docker images to free up space in the runner"
runs:
using: "composite"
steps:
- name: Get initial disk space
run: |
echo "Getting initial free space"
df . -h
shell: bash

- name: Remove language runtimes
run: |
sudo rm -rf \
/usr/share/dotnet /usr/local/lib/android /opt/ghc \
/usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
/usr/lib/jvm /opt/hostedtoolcache/CodeQL || true

echo "Removed language runtimes"
df . -h
shell: bash

- name: Remove docker images
run: |
sudo docker image prune --all --force

echo "Removed docker images"
df . -h
shell: bash
212 changes: 135 additions & 77 deletions .github/workflows/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ env:
EXASOL_VERSION: 7.0.22
NUM_NODES: 2
ETL_TEST_THREADS: 4
ETL_JOB_TIMEOUT: 10
ETL_JOB_TIMEOUT: 20

jobs:
format:
Expand All @@ -21,7 +21,8 @@ jobs:
steps:
- uses: actions/checkout@v3

- uses: actions-rs/toolchain@v1
- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: nightly
Expand Down Expand Up @@ -52,7 +53,8 @@ jobs:
steps:
- uses: actions/checkout@v3

- uses: actions-rs/toolchain@v1
- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
Expand All @@ -68,66 +70,23 @@ jobs:
env:
RUSTFLAGS: -D warnings

tests:
name: Tests
connection_tests:
name: Connection tests
needs: clippy
runs-on: ubuntu-20.04

steps:
- name: Free disk space
run: |
echo "Getting initial free space"
df . -h

sudo rm -rf \
/usr/share/dotnet /usr/local/lib/android /opt/ghc \
/usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
/usr/lib/jvm /opt/hostedtoolcache/CodeQL || true

echo "Removed language runtimes"
df . -h

sudo docker image prune --all --force

echo "Removed docker images"
df . -h

- name: Restore Exasol image
id: cache-docker-exasol
uses: actions/cache@v3
with:
path: ~/ci/cache/docker
key: cache-docker-exasol-${{ env.EXASOL_VERSION }}

- name: Store Exasol image if cache miss
if: steps.cache-docker-exasol.outputs.cache-hit != 'true'
run: docker pull exasol/docker-db:${{ env.EXASOL_VERSION }} && mkdir -p ~/ci/cache/docker && docker image save exasol/docker-db:${{ env.EXASOL_VERSION }} --output ~/ci/cache/docker/exasol-${{ env.EXASOL_VERSION }}.tar

- name: Load Exasol image
run: docker image load --input ~/ci/cache/docker/exasol-${{ env.EXASOL_VERSION }}.tar

- name: Set up Python 3.10
uses: actions/setup-python@v3
with:
python-version: "3.10"
- uses: actions/checkout@v3

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pipenv
- name: Free disk space
uses: ./.github/actions/free-space

- uses: actions/checkout@v3
- name: Create Exasol cluster
id: exa-cluster
uses: ./.github/actions/exa-cluster
with:
repository: Exasol/docker-db
ref: ${{ env.EXASOL_VERSION }}

- name: Set up Exasol Cluster
run: |
pipenv install -r exadt_requirements.txt
pipenv run ./exadt create-cluster --root ~/sqlx --create-root sqlx
pipenv run ./exadt init-cluster --image exasol/docker-db:${{ env.EXASOL_VERSION }} --license ./license/license.xml --num-nodes ${{ env.NUM_NODES }} --auto-storage sqlx
pipenv run ./exadt start-cluster sqlx
# docker update --memory=2g --memory-swap=2g $(docker ps | awk 'NR>1 {print $1}')
exasol-version: ${{ env.EXASOL_VERSION }}
num-nodes: ${{ env.NUM_NODES }}

- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
Expand All @@ -138,59 +97,158 @@ jobs:

- uses: Swatinem/rust-cache@v2

- uses: actions/checkout@v3

# The exadt tool always creates the Exasol cluster with this subnet
- name: Create connection strings
run: |
DATABASE_URL="exa://sys:[email protected]$NUM_NODES:8563"
echo "NO_TLS_URL=$DATABASE_URL?ssl-mode=disabled" >> $GITHUB_ENV
echo "TLS_URL=$DATABASE_URL?ssl-mode=required" >> $GITHUB_ENV

- name: Connection tests
run: cargo test --features migrate,rust_decimal,uuid,chrono -- --nocapture
env:
DATABASE_URL: ${{ env.NO_TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.no-tls-url }}

- name: Connection tests with compression
run: cargo test --features migrate,compression -- --ignored --nocapture
env:
DATABASE_URL: ${{ env.NO_TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.no-tls-url }}

tls_connection_tests:
name: TLS connection tests
needs: clippy
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3

- name: Free disk space
uses: ./.github/actions/free-space

- name: Create Exasol cluster
id: exa-cluster
uses: ./.github/actions/exa-cluster
with:
exasol-version: ${{ env.EXASOL_VERSION }}
num-nodes: ${{ env.NUM_NODES }}

- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: ${{ env.RUST_TOOLCHAIN }}
override: true

- uses: Swatinem/rust-cache@v2

- name: TLS connection tests
run: cargo test --features migrate,rust_decimal,uuid,chrono -- --nocapture
env:
DATABASE_URL: ${{ env.TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}

- name: TLS connection tests with compression
run: cargo test --features migrate,compression -- --ignored --nocapture
env:
DATABASE_URL: ${{ env.TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}

- name: Tests compilation failure if both ETL TLS features are enabled
run: cargo test --features etl_native_tls,etl_rustls || true
env:
DATABASE_URL: ${{ env.TLS_URL }}
etl_tests:
name: ETL tests
needs: clippy
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3

- name: Free disk space
uses: ./.github/actions/free-space

- name: Create Exasol cluster
id: exa-cluster
uses: ./.github/actions/exa-cluster
with:
exasol-version: ${{ env.EXASOL_VERSION }}
num-nodes: ${{ env.NUM_NODES }}

- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: ${{ env.RUST_TOOLCHAIN }}
override: true

- uses: Swatinem/rust-cache@v2

- name: ETL tests
timeout-minutes: ${{ fromJSON(env.ETL_JOB_TIMEOUT) }}
run: cargo test --features migrate,compression,etl -- --ignored --nocapture --test-threads ${{ env.ETL_TEST_THREADS }}
env:
DATABASE_URL: ${{ env.NO_TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.no-tls-url }}

- name: ETL without TLS feature but TLS connection (should fail)
run: cargo test --features migrate,etl -- --ignored --nocapture --test-threads ${{ env.ETL_TEST_THREADS }} || true
env:
DATABASE_URL: ${{ env.TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}

- name: Tests compilation failure if both ETL TLS features are enabled
run: cargo test --features etl_native_tls,etl_rustls || true
env:
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}

native_tls_etl_tests:
name: ETL tests with native-tls
needs: clippy
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3

- name: Free disk space
uses: ./.github/actions/free-space

- name: Create Exasol cluster
id: exa-cluster
uses: ./.github/actions/exa-cluster
with:
exasol-version: ${{ env.EXASOL_VERSION }}
num-nodes: ${{ env.NUM_NODES }}

- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: ${{ env.RUST_TOOLCHAIN }}
override: true

- uses: Swatinem/rust-cache@v2

- name: Native-TLS ETL tests
timeout-minutes: ${{ fromJSON(env.ETL_JOB_TIMEOUT) }}
run: cargo test --features migrate,compression,etl_native_tls -- --ignored --nocapture --test-threads ${{ env.ETL_TEST_THREADS }}
env:
DATABASE_URL: ${{ env.TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}

rustls_etl_tests:
name: ETL tests with rustls
needs: clippy
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3

- name: Free disk space
uses: ./.github/actions/free-space

- name: Create Exasol cluster
id: exa-cluster
uses: ./.github/actions/exa-cluster
with:
exasol-version: ${{ env.EXASOL_VERSION }}
num-nodes: ${{ env.NUM_NODES }}

- name: Setup Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: ${{ env.RUST_TOOLCHAIN }}
override: true

- uses: Swatinem/rust-cache@v2

- name: Rustls ETL tests
timeout-minutes: ${{ fromJSON(env.ETL_JOB_TIMEOUT) }}
run: cargo test --features migrate,compression,etl_rustls -- --ignored --nocapture --test-threads ${{ env.ETL_TEST_THREADS }}
env:
DATABASE_URL: ${{ env.TLS_URL }}
DATABASE_URL: ${{ steps.exa-cluster.outputs.tls-url }}
Loading
Loading