Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(source-gcs): added oauth flow #45414

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

feat(source-gcs): added oauth flow #45414

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
3639a58
added oauth flow
darynaishchenko Sep 12, 2024
07d9a70
added config migrations
darynaishchenko Sep 13, 2024
4433613
added unit tests
darynaishchenko Sep 13, 2024
9f49cd7
bump version
darynaishchenko Sep 13, 2024
328aa7a
Merge branch 'master' into daryna/source-gcs/enable-oauth
darynaishchenko Sep 13, 2024
9545597
Merge branch 'master' into daryna/source-gcs/enable-oauth
darynaishchenko Sep 16, 2024
e8ec691
fixed: changelog, discriminator
darynaishchenko Sep 16, 2024
5cba3bb
Merge branch 'master' into daryna/source-gcs/enable-oauth
darynaishchenko Sep 26, 2024
d306b3c
format fix
darynaishchenko Sep 26, 2024
6e206ad
fix config migrations
darynaishchenko Sep 27, 2024
df3340f
fix spec and legacy config transformations
darynaishchenko Sep 27, 2024
473192d
fix unit tests
darynaishchenko Sep 27, 2024
0598ffe
update uri and open_file
darynaishchenko Oct 15, 2024
6fab547
Merge branch 'master' into daryna/source-gcs/enable-oauth
darynaishchenko Oct 15, 2024
bf51350
fix cat and unit tests
darynaishchenko Oct 15, 2024
50a6bfd
fix unit tests
darynaishchenko Oct 15, 2024
5c262e9
Merge branch 'master' into daryna/source-gcs/enable-oauth
darynaishchenko Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 121 additions & 6 deletions airbyte-integrations/connectors/source-gcs/integration_tests/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,12 +437,83 @@
"required": ["name", "format"]
}
},
"service_account": {
"title": "Service Account Information",
"description": "Enter your Google Cloud <a href=\"https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys\">service account key</a> in JSON format",
"airbyte_secret": true,
"credentials": {
"title": "Authentication",
"description": "Credentials for connecting to the Google Cloud Storage API",
"type": "object",
"order": 0,
"type": "string"
"discriminator": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect the discriminator fields to trip up the Terraform provider. Maybe we need a method similar to this one where we remove them from the schemas

airbyte/airbyte-cdk/python/airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

Lines 58 to 60 in fc12432

def remove_discriminator(schema: Dict[str, Any]) -> None:
"""pydantic adds "discriminator" to the schema for oneOfs, which is not treated right by the platform as we inline all references"""
dpath.delete(schema, "properties/**/discriminator")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed def remove_discriminator from Config class, so both discriminators are being removed from spec

"propertyName": "auth_type",
"mapping": {
"Client": "#/definitions/OAuthCredentials",
"Service": "#/definitions/ServiceAccountCredentials"
}
},
"oneOf": [
{
"title": "Authenticate via Google (OAuth)",
"type": "object",
"properties": {
"auth_type": {
"title": "Auth Type",
"default": "Client",
"const": "Client",
"enum": ["Client"],
"type": "string"
},
"client_id": {
"title": "Client ID",
"description": "Client ID",
"airbyte_secret": true,
"type": "string"
},
"client_secret": {
"title": "Client Secret",
"description": "Client Secret",
"airbyte_secret": true,
"type": "string"
},
"access_token": {
"title": "Access Token",
"description": "Access Token",
"airbyte_secret": true,
"type": "string"
},
"refresh_token": {
"title": "Access Token",
"description": "Access Token",
"airbyte_secret": true,
"type": "string"
}
},
"required": [
"client_id",
"client_secret",
"access_token",
"refresh_token"
]
},
{
"title": "Service Account Authentication.",
"type": "object",
"properties": {
"auth_type": {
"title": "Auth Type",
"default": "Service",
"const": "Service",
"enum": ["Service"],
"type": "string"
},
"service_account": {
"title": "Service Account Information.",
"description": "Enter your Google Cloud <a href=\"https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys\">service account key</a> in JSON format",
"airbyte_secret": true,
"type": "string"
}
},
"required": ["service_account"]
}
]
},
"bucket": {
"title": "Bucket",
Expand All @@ -451,6 +522,50 @@
"type": "string"
}
},
"required": ["streams", "service_account", "bucket"]
"required": ["streams", "credentials", "bucket"]
},
"advanced_auth": {
"auth_flow_type": "oauth2.0",
"predicate_key": ["credentials", "auth_type"],
"predicate_value": "Client",
"oauth_config_specification": {
"complete_oauth_output_specification": {
"type": "object",
"properties": {
"access_token": {
"type": "string",
"path_in_connector_config": ["credentials", "access_token"]
},
"refresh_token": {
"type": "string",
"path_in_connector_config": ["credentials", "refresh_token"]
}
}
},
"complete_oauth_server_input_specification": {
"type": "object",
"properties": {
"client_id": {
"type": "string"
},
"client_secret": {
"type": "string"
}
}
},
"complete_oauth_server_output_specification": {
"type": "object",
"properties": {
"client_id": {
"type": "string",
"path_in_connector_config": ["credentials", "client_id"]
},
"client_secret": {
"type": "string",
"path_in_connector_config": ["credentials", "client_secret"]
}
}
}
}
}
}
3 changes: 2 additions & 1 deletion airbyte-integrations/connectors/source-gcs/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ data:
connectorSubtype: file
connectorType: source
definitionId: 2a8c41ae-8c23-4be0-a73f-2ab10ca1a820
dockerImageTag: 0.6.6
dockerImageTag: 0.7.0
dockerRepository: airbyte/source-gcs
documentationUrl: https://docs.airbyte.com/integrations/sources/gcs
githubIssueLabel: source-gcs
Expand All @@ -29,6 +29,7 @@ data:
- language:python
- cdk:python-file-based
connectorTestSuitesOptions:
- suite: unitTests
- suite: liveTests
testConnections:
- name: gcs_old_config_dev_null
Expand Down
Loading
Loading