-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🪟🐛 [Connector Builder] Fix manifest conversion bugs #21828
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, tested the cases and everything works really well! Just left one nit-pick which can be solved without another review.
The ref thing is interesting, I wasn't aware yaml would do such a thing, TIL.
} else if (manifestAuthenticator.type === "CustomAuthenticator") { | ||
throw new ManifestCompatibilityError(streamName, "uses a CustomAuthenticator"); | ||
} else if (manifestAuthenticator.type === "OAuthAuthenticator") { | ||
if ( | ||
Object.values(manifestAuthenticator.refresh_request_body ?? {}).filter( | ||
(val) => (val !== null && typeof val === "object") || Array.isArray(val) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ultra nit-picky - the only values we support are strings but this logic will let numbers and booleans through. The behavior is the following:
- As long as you don't touch the input, the type is preserved
- As soon as you type in the value field, it will implicitly cast to string
Seems cleaner to just throw on this until we actually support it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense, I agree we should just throw on non-string values for now. Updated
What
When testing out something unrelated, I discovered a few bugs with our YAML -> UI conversion logic, which are fixed in this PR.
How
Bugs fixed:
stream_names
, which caused errors for users when trying to use their generated YAML manifests, as described in [Connector Builder] Setcheck
to the first stream by default #21612convertToManifest
method to filter down to the stream names in the builder form valuescheckStreams
field which match the actual names of streams. If there are no matching stream names, it just sets it to the first stream (unless there are no streams, in which case it sets it to[]
. A connector with no streams is useless anyway so this should be fine).The
manifestToBuilderForm
code already sets thevalues.checkStreams
field to what is set in the manifest;s CheckStream block, so this will use the set of valid stream names from the manifest if present.type
set, we would encounter a runtime error trying to access the type.airbyte/airbyte-integrations/connectors/source-help-scout/source_help_scout/help_scout.yaml
Line 9 in 8b9ddba
This is not desired, as it doesn't match the ref mechanism that is used in the low-code CDK, and it is not expected for us to be producing refs of any form in the YAMLs at the moment.
noRefs
option totrue
in the js-yamldump()
call that we make, which should prevent it from generating refsI also added unit tests for the complex cases above