🪟🎉 Connector builder: Add manual schema (#20862)

* improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <[email protected]> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <[email protected]> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <[email protected]> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <[email protected]> Co-authored-by: lmossman <[email protected]> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <[email protected]> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <[email protected]> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <[email protected]> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <[email protected]> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <[email protected]> * use updated manifest types Co-authored-by: Lake Mossman <[email protected]> * add manual schema * make buttons not submit the form accidentally * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * improve error indicator * make schema editor take up full height (#21000) * prevent resizing card when error message is shown or hidden * Revert "improve error indicator" This reverts commit abb9c0b. Co-authored-by: Tim Roes <[email protected]> Co-authored-by: lmossman <[email protected]>
airbytehq · Jan 5, 2023 · f2f17af · f2f17af
1 parent 2381403
commit f2f17af
Show file tree

Hide file tree

Showing 6 changed files with 172 additions and 49 deletions.
diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/BuilderConfigView.module.scss b/airbyte-webapp/src/components/connectorBuilder/Builder/BuilderConfigView.module.scss
@@ -5,6 +5,7 @@
   display: flex;
   flex-direction: column;
   gap: variables.$spacing-md;
+  height: 100%;
 }
 
 .heading {

diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/BuilderOptional.tsx b/airbyte-webapp/src/components/connectorBuilder/Builder/BuilderOptional.tsx
@@ -11,6 +11,7 @@ export const BuilderOptional: React.FC<React.PropsWithChildren<unknown>> = ({ ch
   return (
     <div className={styles.wrapper}>
       <button
+        type="button"
         onClick={() => {
           setIsOpen(!isOpen);
         }}

diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/StreamConfigView.module.scss b/airbyte-webapp/src/components/connectorBuilder/Builder/StreamConfigView.module.scss
@@ -6,7 +6,8 @@ $controlButtonWidth: 24px;
 .controls {
   display: flex;
   justify-content: center;
-  gap: variables.$spacing-sm;
+  align-items: center;
+  gap: variables.$spacing-md;
 }
 
 .controlButton {
@@ -33,6 +34,33 @@ $controlButtonWidth: 24px;
   }
 }
 
+.tab {
+  background: none;
+  border: none;
+  padding: variables.$spacing-md;
+  border-bottom: variables.$border-thick solid transparent;
+  color: colors.$grey-400;
+  cursor: pointer;
+  display: flex;
+  gap: variables.$spacing-md;
+  align-items: center;
+}
+
+.selectedTab {
+  color: colors.$black;
+  border-bottom-color: colors.$yellow-900;
+}
+
+.errorMessage {
+  // hardcode height to prevent resizing when error message is hidden
+  height: 16px;
+  color: colors.$red;
+}
+
+.schemaEditor {
+  height: 100%;
+}
+
 .multiStreams {
   padding-right: 50px;
 }
diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/StreamConfigView.tsx b/airbyte-webapp/src/components/connectorBuilder/Builder/StreamConfigView.tsx
@@ -2,7 +2,12 @@ import { faTrashCan, faCopy } from "@fortawesome/free-regular-svg-icons";
 import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
 import classNames from "classnames";
 import { useField } from "formik";
-import { useIntl } from "react-intl";
+import { useState } from "react";
+import { FormattedMessage, useIntl } from "react-intl";
+
+import Indicator from "components/Indicator";
+import { CodeEditor } from "components/ui/CodeEditor";
+import { Text } from "components/ui/Text";
 
 import { useConfirmationModalService } from "hooks/services/ConfirmationModal";
 import { BuilderView, useConnectorBuilderState } from "services/connectorBuilder/ConnectorBuilderStateService";
@@ -26,6 +31,7 @@ interface StreamConfigViewProps {
 export const StreamConfigView: React.FC<StreamConfigViewProps> = ({ streamNum, hasMultipleStreams }) => {
   const { formatMessage } = useIntl();
   const [field, , helpers] = useField<BuilderStream[]>("streams");
+  const [selectedTab, setSelectedTab] = useState<"configuration" | "schema">("configuration");
   const { openConfirmationModal, closeConfirmationModal } = useConfirmationModalService();
   const { setSelectedView, setTestStreamIndex } = useConnectorBuilderState();
 
@@ -49,6 +55,9 @@ export const StreamConfigView: React.FC<StreamConfigViewProps> = ({ streamNum, h
     });
   };
 
+  const [, meta] = useField<string | undefined>(streamFieldPath("schema"));
+  const hasSchemaErrors = Boolean(meta.error);
+
   return (
     <BuilderConfigView
       heading={formatMessage({ id: "connectorBuilder.stream" })}
@@ -57,6 +66,17 @@ export const StreamConfigView: React.FC<StreamConfigViewProps> = ({ streamNum, h
       {/* Not using intl for the labels and tooltips in this component in order to keep maintainence simple */}
       <BuilderTitle path={streamFieldPath("name")} label="Stream Name" size="md" />
       <div className={styles.controls}>
+        <StreamTab
+          label={formatMessage({ id: "connectorBuilder.streamConfiguration" })}
+          selected={selectedTab === "configuration"}
+          onSelect={() => setSelectedTab("configuration")}
+        />
+        <StreamTab
+          label={formatMessage({ id: "connectorBuilder.streamSchema" })}
+          selected={selectedTab === "schema"}
+          onSelect={() => setSelectedTab("schema")}
+          showErrorIndicator={hasSchemaErrors}
+        />
         <AddStreamButton
           onAddStream={(addedStreamNum) => {
             setSelectedView(addedStreamNum);
@@ -73,53 +93,96 @@ export const StreamConfigView: React.FC<StreamConfigViewProps> = ({ streamNum, h
           <FontAwesomeIcon icon={faTrashCan} />
         </button>
       </div>
-      <BuilderCard>
-        <BuilderField
-          type="string"
-          path={streamFieldPath("urlPath")}
-          label="Path URL"
-          tooltip="Path of the endpoint that this stream represents."
-        />
-        <BuilderField
-          type="enum"
-          path={streamFieldPath("httpMethod")}
-          options={["GET", "POST"]}
-          label="HTTP Method"
-          tooltip="HTTP method to use for requests sent to the API"
-        />
-        <BuilderField
-          type="array"
-          path={streamFieldPath("fieldPointer")}
-          label="Record selector"
-          tooltip="Pointer into the response that should be extracted as the final record"
-        />
-        <BuilderField
-          type="array"
-          path={streamFieldPath("primaryKey")}
-          label="Primary key"
-          tooltip="Pointer into the response that should be used as the primary key when deduplicating records in the destination"
-          optional
-        />
-      </BuilderCard>
-      <PaginationSection streamFieldPath={streamFieldPath} currentStreamIndex={streamNum} />
-      <StreamSlicerSection streamFieldPath={streamFieldPath} currentStreamIndex={streamNum} />
-      <BuilderCard>
-        <KeyValueListField
-          path={streamFieldPath("requestOptions.requestParameters")}
-          label="Request Parameters"
-          tooltip="Parameters to attach to API requests"
-        />
-        <KeyValueListField
-          path={streamFieldPath("requestOptions.requestHeaders")}
-          label="Request Headers"
-          tooltip="Headers to attach to API requests"
-        />
-        <KeyValueListField
-          path={streamFieldPath("requestOptions.requestBody")}
-          label="Request Body"
-          tooltip="Body to attach to API requests as url-encoded form values"
-        />
-      </BuilderCard>
+      {selectedTab === "configuration" ? (
+        <>
+          <BuilderCard>
+            <BuilderField
+              type="string"
+              path={streamFieldPath("urlPath")}
+              label="Path URL"
+              tooltip="Path of the endpoint that this stream represents."
+            />
+            <BuilderField
+              type="enum"
+              path={streamFieldPath("httpMethod")}
+              options={["GET", "POST"]}
+              label="HTTP Method"
+              tooltip="HTTP method to use for requests sent to the API"
+            />
+            <BuilderField
+              type="array"
+              path={streamFieldPath("fieldPointer")}
+              label="Record selector"
+              tooltip="Pointer into the response that should be extracted as the final record"
+            />
+            <BuilderField
+              type="array"
+              path={streamFieldPath("primaryKey")}
+              label="Primary key"
+              tooltip="Pointer into the response that should be used as the primary key when deduplicating records in the destination"
+              optional
+            />
+          </BuilderCard>
+          <PaginationSection streamFieldPath={streamFieldPath} currentStreamIndex={streamNum} />
+          <StreamSlicerSection streamFieldPath={streamFieldPath} currentStreamIndex={streamNum} />
+          <BuilderCard>
+            <KeyValueListField
+              path={streamFieldPath("requestOptions.requestParameters")}
+              label="Request Parameters"
+              tooltip="Parameters to attach to API requests"
+            />
+            <KeyValueListField
+              path={streamFieldPath("requestOptions.requestHeaders")}
+              label="Request Headers"
+              tooltip="Headers to attach to API requests"
+            />
+            <KeyValueListField
+              path={streamFieldPath("requestOptions.requestBody")}
+              label="Request Body"
+              tooltip="Body to attach to API requests as url-encoded form values"
+            />
+          </BuilderCard>
+        </>
+      ) : (
+        <BuilderCard className={styles.schemaEditor}>
+          <SchemaEditor streamFieldPath={streamFieldPath} />
+        </BuilderCard>
+      )}
     </BuilderConfigView>
   );
 };
+
+const StreamTab = ({
+  selected,
+  label,
+  onSelect,
+  showErrorIndicator,
+}: {
+  selected: boolean;
+  label: string;
+  onSelect: () => void;
+  showErrorIndicator?: boolean;
+}) => (
+  <button type="button" className={classNames(styles.tab, { [styles.selectedTab]: selected })} onClick={onSelect}>
+    {label}
+    {showErrorIndicator && <Indicator />}
+  </button>
+);
+
+const SchemaEditor = ({ streamFieldPath }: { streamFieldPath: (fieldPath: string) => string }) => {
+  const [field, meta, helpers] = useField<string | undefined>(streamFieldPath("schema"));
+
+  return (
+    <>
+      <CodeEditor
+        value={field.value || ""}
+        language="json"
+        theme="airbyte-light"
+        onChange={(val: string | undefined) => {
+          helpers.setValue(val);
+        }}
+      />
+      <Text className={styles.errorMessage}>{meta.error && <FormattedMessage id={meta.error} />}</Text>
+    </>
+  );
+};
diff --git a/airbyte-webapp/src/components/connectorBuilder/types.ts b/airbyte-webapp/src/components/connectorBuilder/types.ts
@@ -66,6 +66,7 @@ export interface BuilderStream {
   };
   paginator?: BuilderPaginator;
   streamSlicer?: SimpleRetrieverStreamSlicer;
+  schema?: string;
 }
 
 export const DEFAULT_BUILDER_FORM_VALUES: BuilderFormValues = {
@@ -274,6 +275,20 @@ export const builderFormValidationSchema = yup.object().shape({
         requestHeaders: yup.array().of(yup.array().of(yup.string())),
         requestBody: yup.array().of(yup.array().of(yup.string())),
       }),
+      schema: yup.string().test({
+        test: (val: string | undefined) => {
+          if (!val) {
+            return true;
+          }
+          try {
+            JSON.parse(val);
+            return true;
+          } catch {
+            return false;
+          }
+        },
+        message: "connectorBuilder.invalidSchema",
+      }),
       paginator: yup
         .object()
         .shape({
@@ -394,12 +409,24 @@ function builderFormAuthenticatorToAuthenticator(
   return globalSettings.authenticator as HttpRequesterAuthenticator;
 }
 
+function parseSchemaString(schema?: string) {
+  if (!schema) {
+    return undefined;
+  }
+  try {
+    return { type: "InlineSchemaLoader", schema: JSON.parse(schema) };
+  } catch {
+    return undefined;
+  }
+}
+
 export const convertToManifest = (values: BuilderFormValues): ConnectorManifest => {
   const manifestStreams: DeclarativeStream[] = values.streams.map((stream) => {
     return {
       type: "DeclarativeStream",
       name: stream.name,
       primary_key: stream.primaryKey,
+      schema_loader: parseSchemaString(stream.schema),
       retriever: {
         type: "SimpleRetriever",
         name: stream.name,

diff --git a/airbyte-webapp/src/locales/en.json b/airbyte-webapp/src/locales/en.json
@@ -718,6 +718,9 @@
   "connectorBuilder.setInUserInput": "This setting is configured as part of the user inputs in the testing panel",
   "connectorBuilder.optionalFieldsLabel": "Optional fields",
   "connectorBuilder.duplicateFieldID": "Make sure no field ID is used multiple times",
+  "connectorBuilder.streamConfiguration": "Configuration",
+  "connectorBuilder.streamSchema": "Schema",
+  "connectorBuilder.invalidSchema": "Invalid JSON - please fix schema to have it applied",
   "connectorBuilder.copyToPaginationTitle": "Copy pagination settings to...",
   "connectorBuilder.copyFromPaginationTitle": "Import pagination settings from...",
   "connectorBuilder.copyToSlicerTitle": "Copy slicing settings to...",