Plan for FE code for Plus in Core

[ssangervasi]

The new challenge that integrating Cls introduces is:

• On a page in the non-Plus Ctl UI, we want to add content that is only usable in the Plus edition.
• This is different from the Data Map, which is an entire page that is only provided by Plus.
• This has some overlap with the nav bar, which needs to have a mix of Plus and non-Plus links.

There are two options to make this possible:

Option 1: The Plus frontend code lives in main repo

With this option, all of the Plus FE is available as open source, but any features which require the Plus BE are disabled.

Pros:

• Quick to get started
• Would work for all similar changes going forward
• FE team gets to centralize in core repo

Cons:

• Any FE code that doesn't rely on BE functionality can't be gated behind Plus.
  • Unless we keep do the Data Map approach and make it a separate zone (page).
• The mismatch Plus FE code that lives in the Plus repo (Data Map) and the main repo could be hard to keep track of.

Alternative considered

Option 2: The Plus frontend builds its own copy of the whole UI

This would require cloning/copying/packaging the Ctl UI code so that Plus can re-use and modify it.

Pros:

• The Plus UI can modify any aspect of the app
• Plus features live in the Plus repo
• Removes zone shenanigans

Cons

• Requires a full rework of how we bring main repo UI code into the Plus repo
• It's likely one repo would start lagging behind the other as we maintain both

[ssangervasi]

Related: https://github.com/ethyca/fidesctl-plus/issues/78

[ssangervasi]

Dug into the design and listed what we'll need to do. I'll convert this list into issues and tackle them roughly in order.

• Initial feature flag for checking Plus status.
  • https://github.com/ethyca/fidesctl-plus/issues/78#issuecomment-1234649352
• Improve feature flag for getting Plus version.
  • Frontend usage of API introduced by Endpoint to tell which version/edition the API is available (plus or not) #1052
• Add "Classify dataset" toggle to Dataset generate form.
  • Issue: Add "Classify dataset" toggle to Dataset generate form #1055
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=20%3A3757
  • With Plus: toggle enabled, defaults to on.
  • Without Plus: toggle disabled.
  • ❓Open questions:
    • Do we want to disable the features, or hide them completely?
    • Need help text for tooltip.
• "Generate dataset" opens a confirmation modal for classify request
  • Issue: Add classification status to Datasets table #1073
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=21%3A3045
  • Clicking "Continue" calls createDataset and generateDataset (just like without classify).
  • Then pass the created dataset to the classify API: https://github.com/ethyca/fidesctl-plus/issues/97
  • When the request succeeds, show a "Success" toast and refresh the list of datasets.
  • ❓Open questions:
    • We do call the Generate API, right? Or is that redundant with Classify?
    • Loading state? Error state?
    • What if creating/generating succeeded, but starting the classify fails? That state would probably be the same as creating a dataset with "classify" turned off.
• Add classificatino status to Datasets table
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A3172
  • Uses the classification status API: https://github.com/ethyca/fidesctl-plus/issues/98
  • Each status has a tooltip description.
    • Processing
    • Awaiting Review
    • Classified
  • ❓Open questions:
    • Without Plus, does is the status column hidden?
    • What is the status if the classifier wasn't used? Is there any idea of review, or expecting the user to add info, as a status?
• Overlay classification details in Dataset table
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A4723
  • Uses the classification details API: https://github.com/ethyca/fidesctl-plus/issues/99
  • The header indicates the classifier has run.
  • The table switcher indicates how many fields have classifier values for each table.
  • The "Identifiability" column contains the classification result.
    • A tooltip indicates the classifier generated the value.
    • If the user assigns a different value, that overrides the classifier.
  • The "Data Categories" column contains the classification results.
    • A tooltip indicates the classifier generated the list of categories.
    • If the adds/removes any categories, those override the classifier.
• Add "Approve dataset classification" button
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A19974
  • Classification status API: https://github.com/ethyca/fidesctl-plus/issues/98#issuecomment-1240033591
  • Before updating the review status, any fields that have a classification result which the user didn't override are saved to the dataset using the updateDataset API.
• Add Datset "Classification" field in editor panel
  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A7634
  • ❓Open questions:
    • Is there an API to assign set this field? Does it live on the dataset model?
    • What even is this?

[mfbrown]

Dug into the design and listed what we'll need to do. I'll convert this list into issues and tackle them roughly in order.

• Initial feature flag for checking Plus status.

  • Shared navigation between Fidesctl and Fidesctl-plus fidesctl-plus#78 (comment)

• Improve feature flag for getting Plus version.

  • Frontend usage of API introduced by Endpoint to tell which version/edition the API is available (plus or not) #1052

• Add "Classify dataset" toggle to Dataset generate form.

  • Issue: Add "Classify dataset" toggle to Dataset generate form #1055

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=20%3A3757

  • With Plus: toggle enabled, defaults to on.

  • Without Plus: toggle disabled.

  • ❓Open questions:

    • Do we want to disable the features, or hide them completely? Hide completely for now. We may eventually give them a treatment to use as a conversion tool for fides-plus but that's not important right now.
    • Need help text for tooltip.

• "Generate dataset" opens a confirmation modal for classify request

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=21%3A3045

  • Clicking "Continue" calls createDataset and generateDataset (just like without classify).

  • Then pass the created dataset to the classify API: Fidesctl-plus endpoint to kick off classification fidesctl-plus#97

  • When the request succeeds, show a "Success" toast and refresh the list of datasets.

  • ❓Open questions:

    • We do call the Generate API, right? Or is that redundant with Classify? @SteveDMurphy @ThomasLaPiana what are your thoughts here? I recall Steve talking about the possibility of doing the classify run and generate simultaneously, in which case we would run generate, but I want to confirm that.
    • Loading state? Error state? @ssangervasi For loading, I think that just is reflected by the "Processing" status on the data set. https://www.figma.com/file/MTbZohIGvB4bq2f2wkiDHO/Fidesops-Management-UI?node-id=3527%3A57671
    • What if creating/generating succeeded, but starting the classify fails? That state would probably be the same as creating a dataset with "classify" turned off. - **I think this works well for now. We'll have to focus on a platform wide error handling/logging solution at some point. **

• Add classificatino status to Datasets table

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A3172

  • Uses the classification status API: Fidesctl-plus endpoint to get classification status fidesctl-plus#98

  • Each status has a tooltip description.

    • Processing
    • Awaiting Review
    • Classified

  • ❓Open questions:

    • Without Plus, does is the status column hidden? I think we can leave the status column in all cases. We can add some statuses which are applied on a manually categorized dataset.
    • What is the status if the classifier wasn't used? Is there any idea of review, or expecting the user to add info, as a status? See above

• Overlay classification details in Dataset table

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A4723

  • Uses the classification details API: Fidesctl-plus endpoint to get classification details fidesctl-plus#99

  • The header indicates the classifier has run.

  • The table switcher indicates how many fields have classifier values for each table.

  • The "Identifiability" column contains the classification result.

    • A tooltip indicates the classifier generated the value.
    • If the user assigns a different value, that overrides the classifier.

  • The "Data Categories" column contains the classification results.

    • A tooltip indicates the classifier generated the list of categories.
    • If the adds/removes any categories, those override the classifier.

• Add "Approve dataset classification" button

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A19974
  • Classification status API: Fidesctl-plus endpoint to get classification status fidesctl-plus#98 (comment)
  • Before updating the review status, any fields that have a classification result which the user didn't override are saved to the dataset using the updateDataset API.

• Add Datset "Classification" field in editor panel

  • Design: https://www.figma.com/file/ArkIrEHQuWeQAQQtuKUXNJ/Classify?node-id=22%3A7634

  • ❓Open questions:

    • Is there an API to assign set this field? Does it live on the dataset model? - This should ultimately be returned by this new endpoint I believe: https://github.com/ethyca/fidesctl-plus/issues/99
    • What even is this? - This is the list of classification suggestions which were generated for the field. Classify will often return more than one suggestion and we want to give users the option to review and select from them if necessary.