Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🪟🔧 Connector builder: Do not rely on local cdk anymore #22391

Merged
merged 7 commits into from
Feb 7, 2023
Merged

🪟🔧 Connector builder: Do not rely on local cdk anymore #22391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
27633c7
do not rely on local cdk anymore
Feb 5, 2023
5e2bc9a
cache versioned schemas
Feb 5, 2023
7b55ca7
fix used version
Feb 5, 2023
69d2ef7
Merge remote-tracking branch 'origin/master' into flash1293/pin-cdk-m…
Feb 6, 2023
65e3daa
Merge branch 'master' into flash1293/pin-cdk-manifest
Feb 6, 2023
6f6c77e
Merge remote-tracking branch 'origin/master' into flash1293/pin-cdk-m…
Feb 7, 2023
2a4019c
review comments
Feb 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-webapp/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"stylelint": "stylelint 'src/**/*.{css,scss}'",
"stylelint-check": "stylelint-config-prettier-scss-check",
"license-check": "node ./scripts/license-check.js",
"generate-client": "orval",
"generate-client": "./scripts/load-declarative-schema.sh && orval",
"validate-links": "ts-node --skip-project ./scripts/validate-links.ts"
},
"dependencies": {
Expand Down
29 changes: 29 additions & 0 deletions airbyte-webapp/scripts/load-declarative-schema.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# This script makes sure the json schema for the low code connector manifest is provided for orval to build the Typescript types
# used by the connector builder UI. It either downloads a released version from PyPI or copies it over from a specified file path.

set -e
flash1293 marked this conversation as resolved.
Show resolved Hide resolved
mkdir -p build

# Make sure this is aligned with the CDK version of the connector builder server
DEFAULT_CDK_VERSION="0.25.0"
Comment on lines +7 to +8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming, is the plan to address keeping these aligned automatically in a follow-up PR, since in the description you said this PR does not fully close that issue?

Do you have an idea of how to solve that?


if [ -z "$CDK_VERSION" ]
then
CDK_VERSION=$DEFAULT_CDK_VERSION
fi


if [ -z "$CDK_MANIFEST_PATH" ]
then
TARGET_FILE="build/declarative_component_schema-${CDK_VERSION}.yaml"
if [ ! -f "$TARGET_FILE" ]; then
echo "Downloading CDK manifest schema $CDK_VERSION from pypi"
curl -L https://pypi.python.org/packages/source/a/airbyte-cdk/airbyte-cdk-${CDK_VERSION}.tar.gz | tar -xzO airbyte-cdk-${CDK_VERSION}/airbyte_cdk/sources/declarative/declarative_component_schema.yaml > ${TARGET_FILE}
else
echo "Found cached CDK manifest schema $CDK_VERSION"
fi
cp ${TARGET_FILE} build/declarative_component_schema.yaml
else
echo "Copying local CDK manifest version from $CDK_MANIFEST_PATH"
cp ${CDK_MANIFEST_PATH} build/declarative_component_schema.yaml
fi
2 changes: 1 addition & 1 deletion airbyte-webapp/src/services/connectorBuilder/connector_manifest_openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ paths: {}
components:
schemas:
ConnectorManifest:
$ref: "../../../../airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml"
$ref: "../../../build/declarative_component_schema.yaml"
9 changes: 9 additions & 0 deletions docs/contributing-to-airbyte/developing-locally.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,15 @@ pnpm start

- Happy Hacking!

#### Using a custom version of the CDK declarative manifest schema for the connector builder UI

When working on the connector builder UI and doing changes to the CDK and the webapp at the same time, you can start the dev server with `CDK_MANIFEST_PATH` or `CDK_VERSION` environment variables set to have the correct Typescript types built. If `CDK_VERSION` is set, it's loading the specified version of the CDK from pypi instead of the default one, if `CDK_MANIFEST_PATH` is set, it's copying the schema file locally.

For example:
```
CDK_MANIFEST_PATH=../../airbyte/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml pnpm start
```

### Connector Specification Caching

The Configuration API caches connector specifications. This is done to avoid needing to run Docker everytime one is needed in the UI. Without this caching, the UI crawls. If you update the specification of a connector and need to clear this cache so the API / UI picks up the change, you have two options:
Expand Down