feat(ingestion) Build out UI form for Snowflake Managed Ingestion #5391

chriscollins3456 · 2022-07-13T21:09:47Z

Builds out a form that we can display in the UI when creating recipes for Managed Ingestion. Right now this PR only introduces it for the Snowflake connector but the plan is to soon introduce this for several more.

The trickiest part of this form is that I've built the ability to switch back and forth between editing a YAML file and editing the form. If you edit values on the form it will update in the yaml file and vice versa. If you remove values from the form, they will be removed from the yaml file. We ultimately still submit the same yaml text payload to the backend, but we're not generating that with the help of a UI form.

Fields are defined as constants with a consistent shape defined by the interface RecipeField. These field objects hold the information for converting its value to the nested javascript object (which is converted to yaml) and from a nested object to a flat Form value. These two methods are abstracted to be shared between all of them and handle fields that also might have parents.

There's plenty of tricky logic around updating values on the Form and in the YAML file and everything showing up correctly on both. For example, if we have an allow pattern for table_patterns and we remove it, we don't want to set allow: [] - instead we want to remove the whole field. Also, if there are no other fields underneath the parent table_patterns then we want to remove the parent as well. It's because of this tricky logic that I wrote a good amount of unit tests on the two setter functions.

Here's what it looks like!

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

github-actions · 2022-07-13T21:36:30Z

Unit Test Results (build & test)

401 tests ±0 401 ✔️ ±0 8m 25s ⏱️ -46s
  96 suites ±0     0 💤 ±0
  96 files ±0     0 ❌ ±0

Results for commit 680956a. ± Comparison against base commit 7bbac5e.

♻️ This comment has been updated with latest results.

gabe-lyons · 2022-07-13T23:32:23Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/constants.ts

@@ -0,0 +1,312 @@
+import { SNOWFLAKE } from '../../conf/snowflake/snowflake';


small nit: should we call this utils.ts instead since it is not just constants?

definitely. it started out as just constants but it of course evolved

gabe-lyons · 2022-07-13T23:38:32Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/constants.ts

+    recipe: any,
+    value: string | boolean | null | undefined,
+    field: string,
+    parentField?: string,


is this assuming that there will be at most 2 layers in the yaml? im concerned that might not always be the case

e.g. profiling.allow_deny_patterns.ignoreCase

or stateful_ingestion.state_provider.type

could we rename parentField to parentPath?

yeah this was something I thought about but didn't build out right away. At first I was trying to keep it simpler to avoid potential bugs from complexity to ensure things work for the only fields we support at the moment (then thinking make more complex / generalized when needed). However I think just making it more generalized for arbitrary number of layers makes sense to just knock out now and be done with.

gabe-lyons · 2022-07-13T23:42:21Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/constants.ts

+export const WAREHOUSE: RecipeField = {
+    name: 'warehouse',
+    label: 'Warehouse',
+    tooltip: 'Snowflake warehouse.',
+    type: FieldType.TEXT,
+    rules: null,
+    getValueFromRecipe: (recipe: any) => recipe.source.config?.warehouse,
+    setValueOnRecipe: (recipe: any, value: string) => setFieldValueOnRecipe(recipe, value, 'warehouse'),
+};


lets namespace these source specific fields by source? e.g. SNOWFLAKE_WAREHOUSE or SNOWFLAKE.WAREHOUSE?

sounds good - especially looking at that tooltip this obviously can't be shared with another connector

gabe-lyons · 2022-07-13T23:43:04Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/constants.ts

+    getValueFromRecipe: (recipe: any) => recipe.source.config?.username,
+    setValueOnRecipe: (recipe: any, value: string) => setFieldValueOnRecipe(recipe, value, 'username'),


it feels like these two lines are redundant. can't we just have a fieldPath property that supplies the fieldpath that these two lambdas both reference?

gabe-lyons · 2022-07-13T23:44:46Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/constants.ts

+
+export const RECIPE_FIELDS = {
+    [SNOWFLAKE]: {
+        fields: [ACCOUNT_ID, WAREHOUSE, USERNAME, PASSWORD, ROLE],
+        advancedFields: [
+            INCLUDE_LINEAGE,
+            IGNORE_START_TIME_LINEAGE,
+            CHECK_ROLE_GRANTS,
+            PROFILING_ENABLED,
+            STATEFUL_INGESTION_ENABLED,
+        ],
+        filterFields: [
+            TABLE_ALLOW,
+            TABLE_DENY,
+            DATABASE_ALLOW,
+            DATABASE_DENY,
+            SCHEMA_ALLOW,
+            SCHEMA_DENY,
+            VIEW_ALLOW,
+            VIEW_DENY,
+        ],
+    },
+};
+


so cool how simple this is

very easy to add / remove fields or whole new connectors now!

gabe-lyons · 2022-07-14T00:00:17Z

datahub-web-react/src/app/ingest/source/builder/DefineRecipeStep.tsx

+// import YAML from 'yamljs';
+// import Form from '@rjsf/core';


nit, can remove

gabe-lyons · 2022-07-14T00:00:42Z

datahub-web-react/src/app/ingest/source/builder/DefineRecipeStep.tsx

@@ -68,6 +74,7 @@ export const DefineRecipeStep = ({ state, updateState, goTo, prev }: StepProps)
        let recipeJson;
        try {
            recipeJson = yamlToJson(stagedRecipeYml);
+            console.log('recipeJson', recipeJson);


gabe-lyons · 2022-07-14T00:01:47Z

datahub-web-react/src/app/ingest/source/builder/RecipeBuilder.tsx

+            YAML.parse(displayRecipe);
+            setIsViewingForm(isFormView);
+        } catch (e) {
+            message.warn('Found invalid YAML. Please fix your recipe in order to switch views.');


do we not get an informative error here?

it looks like we actually do! I was snagging this from elsewhere since this is what we were doing but we do get the problem line - will update

gabe-lyons · 2022-07-14T00:03:09Z

datahub-web-react/src/app/ingest/source/builder/RecipeForm/RecipeForm.tsx

+        allFields.forEach((field) => {
+            updatedValues = field.setValueOnRecipe(updatedValues, allValues[field.name]);
+        });


couldn't we just iterate over _changedValues? would that be faster?

why yes it would

gabe-lyons · 2022-07-14T21:32:08Z

datahub-web-react/src/app/ingest/source/builder/RecipeBuilder.tsx

+            const messageText = (e as any).parsedLine
+                ? `Fix line ${(e as any).parsedLine} in your recipe`
+                : 'Please fix your recipe';
+            message.warn(`Found invalid YAML. ${messageText} in order to switch views.`);


…tahub-project#5391)

* feat(ingest): working with multiple bigquery projects (datahub-project#5240) * fix(build): missing libs (datahub-project#5254) * fix(build): use correct creds (datahub-project#5261) * feat(ingest): Option to define path spec for Redshift lineage generation (datahub-project#5256) * fix(ui): Enable previews properly when browsing for DataJob (datahub-project#5250) * fix(docs): Fix acronym on mxe docs (datahub-project#5249) * fix(ui): Support deleting references to glossary terms / nodes, users, assertions, and groups (datahub-project#5248) * Adding referential integrity to deletes API * Updating comments * Fix build * fix checkstyle * Fixing Delete Entity utils Test * feat(docs) add links in quickstart for adding users (datahub-project#5267) * fix(siblings) Display sibling assertions in Validations tab (datahub-project#5268) * fix(siblings) Display sibling assertions in Validations tab * query changes Co-authored-by: Chris Collins <[email protected]> * feat(domain) Add ability to edit a Domain name from the UI (datahub-project#5266) * feat(ingest): delta-lake: adding support for delta lake (datahub-project#5259) Co-authored-by: Shirshanka Das <[email protected]> * fix(siblings) Update the names of siblings utils args for readability (datahub-project#5269) Co-authored-by: Chris Collins <[email protected]> * docs(adopters): add showroomprive and n26 as DataHub adopters (datahub-project#5271) * feat(glossary) Add Source section to sidebar for Glossary Terms (datahub-project#5262) * fix(ingest): delta-lake - fix dependency issue for snowflake due to s3_util (datahub-project#5274) * fix(ingest): s3 - Remove unneeded methods from s3_util (datahub-project#5276) * feat(ui): Selector recommendations in Owner, Tag and Domain Modal (datahub-project#5197) * fix(security) Sanitize rich text before sending to backend or rendering on frontend (datahub-project#5278) * feat(GraphQL): Support for Deleting Domains, Tags via GraphQL API (datahub-project#5272) * feat(build): reduce build time for ingestion image (datahub-project#5225) * fix(ingestion): profiling - Fixing partitioned table profiling in BQ (datahub-project#5283) * fix(ingest) redshift: Adding missing dependencies and relaxing sqlalchemy dependency (datahub-project#5284) Relaxing sqlalchemy deps to make our plugins work with Airflow 2.3 * fix(ingestion): Reverting sqlalchemy upgrade because it caused issues with mssql and redshift-usage (datahub-project#5289) * fix(Siblings): Have sibling hook use entity client (datahub-project#5279) * fixing dbt platform issues * have sibling hook use entity client over entity service * switching search service as well * lint * more lint * more specific exceptions * refactor(ui): Show message when related glossary terms are empty. (datahub-project#5285) * docs(adopter): add Digital Turbine as DataHub adopter (datahub-project#5290) * docs(docker): Update schema-registry docker.env (datahub-project#5231) * feat(siblings): index sibling aspects for historical dbt metadata (datahub-project#5291) * fixing dbt platform issues * starting sibling restore index job work * finish restore indices * migrating to list urns * rename constant * disaster recovery * feat(ui) Adding support for deleting Tags and Domains via the UI (datahub-project#5280) * Adding support for deleting tags and domains via the UI * Fixing tests * fix(test): add cleanup in tests, make urls configurable (datahub-project#5287) * fix(docs,quickstart): release related changes for 0.8.40 (datahub-project#5299) * fix(doc): config typo on confluent cloud doc (datahub-project#5293) * fix(cli): suppress secrets in stacktraces (datahub-project#5302) * Minor UI bug fuix (datahub-project#5292) * fix(cli): timeline - category should be owner not ownership (datahub-project#5304) * perf(ui): reduce data fetched by siblings in lineage (datahub-project#5308) * fix(ingest): bigquery - Fix for bigquery error when there was no bigquery catalog specified (datahub-project#5303) * fix(ui) Fix entity profile sidebar width issues (datahub-project#5305) Co-authored-by: Chris Collins <[email protected]> * perf(search): Improve search default performance (datahub-project#5311) * perf(ui): Performance improvements and misc refactorings in the UI (datahub-project#5310) * feat(ui): Modified the drop down of Menu Items (datahub-project#5301) * fix(validation) Fail validation error silently instead of crashing (datahub-project#5314) * feat(docs) Add documentation on authorization & authentication (datahub-project#5265) * fix(ui) Make profile icon clickable to expand header menu (datahub-project#5317) * refactor(ui): Extract searchable page into its own component (perf + ux) (datahub-project#5318) * fix(gms) Remove auto-creating status aspects if not present when ingesting metadata (datahub-project#5315) * fix(ui): Add missing SearchRoutes component (datahub-project#5321) * feat(ingest): looker - ingest dashboard create/update/delete timestamps (datahub-project#5312) * fix(ui): Fix pipeline tasks list loading (datahub-project#5332) * feat(ingest): lookml - adding support for only emitting reachable views from explores (datahub-project#5333) * fix(ingest): tableau - omit schema fields when name is absent (datahub-project#5275) * fix(siblings) Combine siblings data but remove duplicate data (datahub-project#5337) * fix(docs): Fix typo in metadata-ingestion.md (datahub-project#5338) * fix(me) Cache the me query for performance reasons (datahub-project#5316) * fix(tokens) Adds non-admin tests for access tokens (datahub-project#5174) * feat(bigquery): support size, rowcount, lastmodified based table selection for profiling (datahub-project#5329) Co-authored-by: Shirshanka Das <[email protected]> * chore: Refactor Python Codebase (datahub-project#5113) * docs(bigquery): profiling report enhancement (datahub-project#5342) * feat(ingest): update CSV source to support description and ownership type (datahub-project#5346) * fix(ui): fixed the ui issue (datahub-project#5341) * feat(ingest): salesforce - add connector (datahub-project#5104) Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Vincent Koc <[email protected]> * feat(bootstrap): create abstract class UpgradeStep to abstract away upgrade logic (datahub-project#5349) * fix(ingest): bigquery-usage - dataset name for sharded tables (datahub-project#5347) Co-authored-by: Shirshanka Das <[email protected]> * docs(features): update grammar (datahub-project#5350) * fix(ci): fix mysql test and attempt kafka-connect ingestion (datahub-project#5352) * feat(ui): add copy function for stats table sample value (datahub-project#5331) * fix(ui) Correct show/hide tabs in Settings based on privileges (datahub-project#5355) Co-authored-by: Chris Collins <[email protected]> * fix(siblings): add useMutationUrn to domain section (datahub-project#5270) * fixing dbt platform issues * useMutationUrn for domains modal * feat(schema) Show last observed timestamp in the schema tab (datahub-project#5348) * fix(glossary) Fixes a bug for yaml ingested terms without source_url (datahub-project#5356) * feat(lineage) Add Lineage tab to Chart and Dashboard entity profiles (datahub-project#5357) * fix(cassandra): fix Cassandra queries used by IngestDataPlatformInstancesStep (datahub-project#5199) * refactor(ui): Use createTag mutation for creating new tags from the UI (datahub-project#5359) * feat(ui): Added recommendation on group modal (datahub-project#5362) * refactor(ui): Remove unnecessary fields in GraphQL (datahub-project#5358) * feat(ingest) - add audit actor urn to auditStamp (datahub-project#5264) * feat(ingest): improve domain ingestion usability (datahub-project#5366) * fix(config): fixes config key in DataHubAuthorizerFactory (datahub-project#5371) * fix(ingest): domains - check whether urn based domain exists during resolution (datahub-project#5373) * feat(quickstart): Adding env variables and cli options for customizing mapped ports in quickstart (datahub-project#5353) Co-authored-by: Shirshanka Das <[email protected]> * fix(build): tweak ingestion build (datahub-project#5374) * feat(sdk): python - add get_aspects_for_entity (datahub-project#5255) Co-authored-by: Shirshanka Das <[email protected]> * fix(airflow): fix for failing serialisation when Param was specified + support for external task sensor (datahub-project#5368) fixes datahub-project#4546 * fix(users): fix to not get invite token unless the invite token modal is visible (datahub-project#5380) * fix(gms) Propagate cache exception upstream (datahub-project#5381) * fix(bootstrap): skip ingesting data platforms that already exist (datahub-project#5382) * fix(cli): respect server telemetry settings correctly (datahub-project#5384) Co-authored-by: Shirshanka Das <[email protected]> * fix(ingest): bigquery - Graceful bq partition id date parsing failure (datahub-project#5386) * feat(airflow): Circuit breaker and python api for Assertion and Operation (datahub-project#5196) * feat(kafka-setup): add options for sasl_plaintext (datahub-project#5385) allow sasl_plaintext options using environment variables * fix(bigquery): multi-project GCP setup run query through correct project (datahub-project#5393) * fix(bigquery): add storage project name (datahub-project#5395) * Add Changes to support smoke test on Datahub deployed on kubernetes Cluster (datahub-project#5334) Co-authored-by: Aseem Bansal <[email protected]> * fix(PlayCookie) PLAY_TOKEN cookie rejected because userprofile exceeds 4096 chars (datahub-project#5114) * feat(dashboards): add datasets field to DashboardInfo aspect (datahub-project#5188) Co-authored-by: John Joyce <[email protected]> * feat(siblings): allow viewing siblings separately (datahub-project#5390) * allow pulling back curtain for siblings * sibling pullback working for lineage + property merge * propagating provinence to ui * fixups from merge & some renames * fix styling & add tooltip * adding cypress tests * fix lint * updating mocks * updating smoke test * fixing domains smoke test * responding to comments * refactor(ui): Added Cursor pointer to tags (datahub-project#5389) * feat(GMS): Adding Dashboard Usage Models (datahub-project#5399) * fix(quickstart): use platform agnostic way to get folder (datahub-project#5400) * Adds support for Domains in CSV source (datahub-project#5372) * feat(ingestion) Build out UI form for Snowflake Managed Ingestion (datahub-project#5391) * fix(kafka): add missing configs (datahub-project#5394) * feat(model): dashboard usage model, is_null condition added (datahub-project#5397) * fix(datahub-client): Fix kafka config issue (datahub-project#5403) * build: improve comprehensiveness of gradle clean (datahub-project#5003) * fix(gms): Change MessageDigest to be thread safe (datahub-project#5405) * fix(metadata-ingestion) Fix broken csv enricher test (datahub-project#5406) * fix(tests): Removes duplicate policies tests & makes DataHub user configurable (datahub-project#5365) * feat(quickstart,docs): updates for v0.8.41 (datahub-project#5409) * fix(ingest): ensure upgrade checks run async (datahub-project#5383) * fix(ingest): looker - pass transport options to all api calls (datahub-project#5417) * feat(quickstart): moving to official confluent images for m1 (datahub-project#5416) * fix(documentation) Fix erratic cursor in documentation editor bug (datahub-project#5411) Co-authored-by: Chris Collins <[email protected]> * feat(ui): Supporting enriched search preview + misc improvements (datahub-project#5419) * chore: remove unnecessary modules from codebase (datahub-project#5420) * fix(ingest): looker - extract usage for dashboards allowed by pattern (datahub-project#5424) * fix(docker): fix kafka-setup command to support same capabilities as previous (datahub-project#5428) * fix(protobuf) Set undeprecated ownership type & fix case sentitive urn corpGroup (datahub-project#5425) * fix(ui): add dataset qualifiedName parameter to lineage query (datahub-project#5427) * fix(glossary) Fix dropdown where disabled buttons are still clickable (datahub-project#5430) Co-authored-by: Chris Collins <[email protected]> * docs(bigquery): add changelog and unittest for profiling limits (datahub-project#5407) * fix(siblings): fixing lineage fetching for siblings & sources (datahub-project#5415) * fix(ui): Fixing unreleased search preview bugs (datahub-project#5432) * feat(ui): Adding Statistics Summary to Dataset + Dashboard Profiles (datahub-project#5440) * feat(ingest): add test source connection feature, structured report file (datahub-project#5442) * fix(ingest/glue): handle error when generating s3 tags for virtual view tables (datahub-project#5398) Co-authored-by: Tim Costa <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> * feat(ingest): model - adding a small extension to support communicating structured responses (datahub-project#5429) * fix(ingest): bigquery-usage - fix dataset name for sharded table (datahub-project#5412) * feat(ingestion) Add new endpoint to test an ingestion connection (datahub-project#5438) * feat(cli,build): remove deprecated variables GMS_HOST/_PORT (datahub-project#5451) * fix(search): make filters by default an empty list if null (datahub-project#5454) * fix(ingest): hive - add column comment as a column description (datahub-project#5449) * feat(groups): add native groups concept to DataHub (datahub-project#5443) * fix(ingest): fix serialization of report to handle nesting (datahub-project#5455) * fix(ingest): tableau - fix tableau db error, add more logs (datahub-project#5423) * build(deps): bump terser from 5.9.0 to 5.14.2 in /docs-website (datahub-project#5448) Bumps [terser](https://github.com/terser/terser) from 5.9.0 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: spark-lineage - configuration details for Amazon EMR (datahub-project#5459) * feat(app): schema-history - remove blame language for the schema history feature (datahub-project#5457) * Worked on the alignment of menu icon in search header (datahub-project#5458) * build(deps): bump terser from 4.8.0 to 4.8.1 in /datahub-web-react (datahub-project#5446) Bumps [terser](https://github.com/terser/terser) from 4.8.0 to 4.8.1. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(ingest): snowflake - basic test connection capability (datahub-project#5464) * fix(ingest/trino): Avoid exception if $properties table empty or not readable (datahub-project#5447) Under some configuration of access rules in Trino, the user may not have read access to the content of the table, which will result in an exception (`fetchone()` returns `None`) This commit ensures no exception are raised and the ingestion can proceed. * feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed (datahub-project#5435) * fix(build): add base image with gradle wrapper cached (datahub-project#5467) * doc(bigquery): groups grants by requirements (datahub-project#5468) * fix(docs,build): remove base image not needed, cleanup docs (datahub-project#5469) * feat(ui): Partial support for Chart usage (datahub-project#5473) * fix(ingest): bigquery: multiproject profiling fix (datahub-project#5474) * fix(ingest): kafka - revert deps back to < 1.9.0 (datahub-project#5476) * feat(ci): datahub-upgrade - support multiplatform image (datahub-project#5477) * feat(cli): quickstart - experimental support for backup restore (datahub-project#5418) * feat(ingest): dbt - updating source lineage logic (datahub-project#5414) Co-authored-by: Shirshanka Das <[email protected]> * Ingestion: Added form in Big Query type to edit the queries. (datahub-project#5431) * docs(reindex): fix docsearch config (datahub-project#5479) * refactor(ui): Adding checkbox option to select multiple results at once. (datahub-project#5422) * feat(cli): delete - hard delete deletes soft deleted entities (datahub-project#5478) * fix(docs): add missing closing marker for note section (datahub-project#5480) * fix(ci): intermittent failure in github actions (datahub-project#5452) * feat(model, ingest): add user email in dashboard user usage counts (datahub-project#5471) * feat(ingest): snowflake - test_connection add support for capability report (datahub-project#5472) Co-authored-by: Shirshanka Das <[email protected]> * feat(build): automatically mark issues as stale to close inactive issues (datahub-project#5482) * fix(ingest): loosen confluent-kafka dep requirement (datahub-project#5489) * refactor(ingest): cleanup importlib.import_module calls (datahub-project#5490) * build(ingest): make gradle build less chatty (datahub-project#5491) * fix(ingest): dbt - add support for trino datatypes (datahub-project#5379) * refactor(ci): use custom action for checking codegen status (datahub-project#5493) * feat(spark-lineage, java-emitter): Support ssl cert disable verification functionality (datahub-project#5488) Co-authored-by: Shirshanka Das <[email protected]> * docs(auth): fix link to point to new doc (datahub-project#5501) * docs(updating-datahub): add note for breaking change in looker usage ingestion (datahub-project#5499) * fix(ingest): cleanup unused flake8 noqa statements (datahub-project#5492) * fix(ingest): cleanup unused flake8 noqa statements In the future, we can discover these using `flake8-noqa`. * add back c901 * refactor(ci): refactor Docker build-and-push workflows (datahub-project#5494) * docs(slack): update to Slack guidelines (datahub-project#5504) * feat(cli): delete - add --only-soft-deleted option, perf improvements (datahub-project#5485) * fix(ingest): use temp dir for file generated during test (datahub-project#5505) * feat(ui) Show Glossary and Domains header links to everyone (datahub-project#5506) Co-authored-by: Chris Collins <[email protected]> * fix(ui): Fix Flickering Issue on search input field (datahub-project#5503) * fix(ingest): respect rest emitter timeout setting (datahub-project#5508) * fix(ui): Flickering Issue on search input field (datahub-project#5515) * feat(ui): Added form to Looker and Tableau (datahub-project#5487) * feat(identity): update azure and okta connectors to emit Origin aspects (datahub-project#5495) * feat(ui): Adding Search Select feature(frontend only) (datahub-project#5507) * test(ingest): limit GMS retries in test (datahub-project#5509) * fix(ingest): airflow: update subdag check for compatibility with older Airflow versions (datahub-project#5523) * use getattr to default None if no subdag * add None check * add other None check * Apply suggestions from code review- double quotes Co-authored-by: Harshal Sheth <[email protected]> * minor tweak to fix lint Co-authored-by: Harshal Sheth <[email protected]> * fix(ingest): fix unbound variable bug in cli ingest list-runs (datahub-project#5527) * fix(ui) Display Term Group name properly in Recently Viewed (datahub-project#5528) * feat(ingestion) Add frontend connection test for Snowflake (datahub-project#5520) * fix(glossary) Fix Glossary success messages and sort Glossary (datahub-project#5533) * show error and success messages in glossary properly * sort glossary nodes and terms alphabetically Co-authored-by: Chris Collins <[email protected]> * feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms (datahub-project#4999) * feat(ingest): add deprecation warning for Python 3.6 (datahub-project#5519) * docs(townhall) add past townhall agendas (datahub-project#5536) * feat(ingestion): add groups to ldap users (datahub-project#5470) * chore(issues): reduce time for issues to be marked stale and then closed (datahub-project#5537) * fix(ingestion) Set pipeline_name on UI recipes with forms (datahub-project#5535) * Fixing OIDC logout issues (datahub-project#5538) * fix(analytics-tab) - fix analytics tab config variable for gms (datahub-project#5529) * feat(ui): Support batch adding / remove tags from search lists. (Batch Actions part 2/7) (datahub-project#5534) * fix(ingestionSource): improve error experience when ingestion source is in an inconsistent state (datahub-project#5522) * fix(docs): Fixed typo in schema history markdown! (datahub-project#5545) * fix(docker): Fixing dev docker and quickstart (datahub-project#5550) * feat(ui): Support Batch adding and removing Glossary Terms (Batch Actions 3/7) (datahub-project#5544) * feat(ci): test quickstart works (datahub-project#5518) * feat(ci): test quickstart works * do not fail fast * remove macos * add some debug information * tweak triggers * fix workflow file * remove running on every PR * Update .github/workflows/check-quickstart.yml Co-authored-by: Harshal Sheth <[email protected]> * Update .github/workflows/check-quickstart.yml Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * test(ingest): mark trino/hana tests as xfail due to flakes (datahub-project#5549) * feat(ingestion): superset - add display_uri to config (datahub-project#5408) * fix(quickstart): failure on a path not being present (datahub-project#5554) * fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests (datahub-project#5540) * fix(dbt): fix issue of dbt stateful ingestion with tests Co-authored-by: MugdhaHardikar-GSLab <[email protected]> Co-authored-by: MohdSiddique Bagwan <[email protected]> Co-authored-by: Ravindra Lanka <[email protected]> * feat(ui): Batch add & remove Owners to assets via the UI (datahub-project#5552) * feat(ingestion) Update managed ingestion scheduler to be easier to use (datahub-project#5559) * fix(ingestion): correct trino datatype handling (datahub-project#5541) Co-authored-by: Ravindra Lanka <[email protected]> * feat(ingest) Allow ingestion of Elasticsearch index template (datahub-project#5444) Co-authored-by: Ravindra Lanka <[email protected]> * fix(ingest): fix some typos and logging issues (datahub-project#5564) * feat(transformers): Add domain transformer for dataset (datahub-project#5456) Co-authored-by: MohdSiddique Bagwan <[email protected]> * chore(0.8.42): update breaking changes doc (datahub-project#5563) * fix(ingest): activate mypy support for ParamSpec typing annotation (datahub-project#5551) * (chore): upgrading ingestion to 0.8.42 (datahub-project#5562) * fix(gms): ensure directory is present (datahub-project#5568) * fix(ci): flaky smoke test fix (datahub-project#5569) * fix(gms): missing directory for gms (datahub-project#5570) * chore(build): tweak stale issue timing (datahub-project#5571) * feat(ui): Batch set & unset Domain for assets via the UI (datahub-project#5560) * extending assertion std model (datahub-project#5575) * feat(ui): Support batch deprecation from the UI (Batch actions part 6/7) (datahub-project#5572) * feat(graphql): add MutableTypeBatchResolver (datahub-project#4976) * feat(ingestion) Implement secrets in new managed ingestion form (datahub-project#5574) * fix(ui): Fixing batch set domains bug (datahub-project#5580) * chore(gradle): update node version for docs site (datahub-project#5581) * feat(test): add read-only smoke tests (datahub-project#5558) * feat(ingestion) Add Save & Run button to managed ingestion builder (datahub-project#5579) * fix(ingest): handle when current server version is unavailable (datahub-project#5547) * feat(ingest): dbt - control over emitting test_results, test_definitions, etc. (datahub-project#5328) Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> * feat(datahub-client): add java file emitter (datahub-project#5578) Co-authored-by: Shirshanka Das <[email protected]> * feat(ingest): infer aspectName from aspect type in MCP (datahub-project#5566) * fix(ingest): sql-common - db2, snowflake bug fixes to extract table descriptions (datahub-project#5526) Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> * fix(ingest): moving delta-lake connector to be 3.7+ only (datahub-project#5584) * feat(ingest): delta-lake - extract table history into operation aspect (datahub-project#5277) Co-authored-by: Shirshanka Das <[email protected]> * fix apache ranger plugin readme file rendering (datahub-project#5585) * feat(ui): make container description searchable and have description show up in results (datahub-project#5586) * fix(groups): fix user, search, and preview group membership to be fetched for both external and native group memberships (datahub-project#5587) * feat(ingest): power-bi - make ownership ingestion optional (datahub-project#5335) Co-authored-by: MohdSiddique Bagwan <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * Expose catalog_name in athena.py (datahub-project#5548) * expose catalog_name to the sql alchemy uri that is passed into pyathena Co-authored-by: Ravindra Lanka <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> * Fix profiling when using {table}. (datahub-project#5531) * profiling fix for when using {table} Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Ravindra Lanka <[email protected]> * feat(ui): Support batch deleting from ui (datahub-project#5582) * feat(ingest): clickhouse - add metadata modification time and data size (datahub-project#5330) Co-authored-by: Ravindra Lanka <[email protected]> * feat(ui): Add rich UI ingestion run summary (datahub-project#5577) * fix(ci): smoke test less flaky, add src, dev dep in smoke image (datahub-project#5594) * updated mock custom to pass the test suite * added env for mysql-setup for smoketest to pass * added env for mysql-setup for smoketest to pass * added env for mysql-setup for smoketest to pass * push to heruko repo instead of linkedin Co-authored-by: Aseem Bansal <[email protected]> Co-authored-by: Tamas Nemeth <[email protected]> Co-authored-by: Michael A. Schlosser <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: Mugdha Hardikar <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: Maggie Hays <[email protected]> Co-authored-by: Ankit keshari <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: liyuhui666 <[email protected]> Co-authored-by: Tengis Batsaikhan <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: Mayuri Nehate <[email protected]> Co-authored-by: dougpm <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Aditya Radhakrishnan <[email protected]> Co-authored-by: Amanda Ng <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: Justin Marozas <[email protected]> Co-authored-by: Sergio Gómez Villamor <[email protected]> Co-authored-by: Navin Sharma <[email protected]> Co-authored-by: Aezo <[email protected]> Co-authored-by: abiwill <[email protected]> Co-authored-by: Felix Lüdin <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: Chris Collins <[email protected]> Co-authored-by: leifker <[email protected]> Co-authored-by: Alexey Kravtsov <[email protected]> Co-authored-by: Tim Costa <[email protected]> Co-authored-by: Tim Costa <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Guillaume Gardey <[email protected]> Co-authored-by: Vishal Shah <[email protected]> Co-authored-by: mohdsiddique <[email protected]> Co-authored-by: Salih Can <[email protected]> Co-authored-by: RyanHolstien <[email protected]> Co-authored-by: Skyler Sinclair <[email protected]> Co-authored-by: Dan Andreescu <[email protected]> Co-authored-by: MohdSiddique Bagwan <[email protected]> Co-authored-by: Ravindra Lanka <[email protected]> Co-authored-by: Marcin Szymański <[email protected]> Co-authored-by: xiphl <[email protected]> Co-authored-by: NoahFournier <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Jordan Wolinsky <[email protected]>

Chris Collins and others added 11 commits July 13, 2022 16:57

have it sloppily working with all types

13fa797

clean up the code a good amount

d215340

make overall styling improvements

1886276

do some file refactoring

948e059

add two sections and style some stuff

cbbc4f1

add most of the fields

10c522c

refactor constants file to get more reuse going

1d5b8f4

most of finishing touches except filter sections

3f75131

add filters sections and help tooltips next to filter fields

8d324b6

fix some edge case bugs and update default yaml for snowflake

ffbb728

final cleanup and add tests

b09ba10

gabe-lyons reviewed Jul 13, 2022

View reviewed changes

gabe-lyons reviewed Jul 14, 2022

View reviewed changes

remove cruft

7847534

maggiehays added the product PR or Issue related to the DataHub UI/UX label Jul 14, 2022

Chris Collins added 3 commits July 14, 2022 16:51

make recipe setter extensible with arbitrary field path

c342b5a

final PR edits

def52fc

fix test titles

ee94aa1

gabe-lyons reviewed Jul 14, 2022

View reviewed changes

add override fields

680956a

gabe-lyons approved these changes Jul 14, 2022

View reviewed changes

anshbansal merged commit fa07dc6 into datahub-project:master Jul 15, 2022

Santhin pushed a commit to Santhin/datahub that referenced this pull request Jul 26, 2022

feat(ingestion) Build out UI form for Snowflake Managed Ingestion (da…

39ca62e

…tahub-project#5391)

maggiehays pushed a commit to maggiehays/datahub that referenced this pull request Aug 1, 2022

feat(ingestion) Build out UI form for Snowflake Managed Ingestion (da…

fe17418

…tahub-project#5391)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingestion) Build out UI form for Snowflake Managed Ingestion #5391

feat(ingestion) Build out UI form for Snowflake Managed Ingestion #5391

chriscollins3456 commented Jul 13, 2022

github-actions bot commented Jul 13, 2022 •

edited

Loading

gabe-lyons Jul 13, 2022

chriscollins3456 Jul 14, 2022

gabe-lyons Jul 13, 2022

gabe-lyons Jul 13, 2022

gabe-lyons Jul 13, 2022

gabe-lyons Jul 13, 2022

chriscollins3456 Jul 14, 2022

gabe-lyons Jul 13, 2022

chriscollins3456 Jul 14, 2022

gabe-lyons Jul 13, 2022

gabe-lyons Jul 13, 2022

chriscollins3456 Jul 14, 2022

gabe-lyons Jul 14, 2022

gabe-lyons Jul 14, 2022

gabe-lyons Jul 14, 2022

chriscollins3456 Jul 14, 2022 •

edited

Loading

gabe-lyons Jul 14, 2022

chriscollins3456 Jul 14, 2022

gabe-lyons Jul 14, 2022

		@@ -0,0 +1,312 @@
		import { SNOWFLAKE } from '../../conf/snowflake/snowflake';

		getValueFromRecipe: (recipe: any) => recipe.source.config?.username,
		setValueOnRecipe: (recipe: any, value: string) => setFieldValueOnRecipe(recipe, value, 'username'),

		// import YAML from 'yamljs';
		// import Form from '@rjsf/core';

feat(ingestion) Build out UI form for Snowflake Managed Ingestion #5391

feat(ingestion) Build out UI form for Snowflake Managed Ingestion #5391

Conversation

chriscollins3456 commented Jul 13, 2022

Checklist

github-actions bot commented Jul 13, 2022 • edited Loading

Unit Test Results (build & test)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriscollins3456 Jul 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 13, 2022 •

edited

Loading

chriscollins3456 Jul 14, 2022 •

edited

Loading