Google Cloud SQL MySQL Connector #4949

andres-torres-marroquin · 2024-06-04T13:57:30Z

Closes PROD-2152

Description Of Changes

Added GoogleCloudSQLMySQLConnector and tests for it.

Code Changes

Created GoogleCloudSQLMySQLConnector
Added tests for GoogleCloudSQLMySQLConnector

Steps to Confirm

Add a system
Add an Integration, select Google Cloud SQL for MySQL
Add testing credentials
Press Test Integration
Add testing Dataset
Submit an access request in the Privacy Center

Pre-Merge Checklist

vercel · 2024-06-04T13:57:34Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Jun 19, 2024 0:52am

tests/ops/integration_tests/test_external_database_connections.py

clients/admin-ui/cypress/fixtures/connectors/connection_types.json

galvana · 2024-06-04T17:00:25Z

src/fides/api/schemas/connection_configuration/connection_secrets_google_cloud_mysql.py

+)
+
+
+class GoogleCloudMySQLSchema(ConnectionConfigSecretsSchema):


This should be using Google's keyfile_creds similar to how we do it in connection_secrets_bigquery. Take a look at option two in this doc for how to authenticate using a keyfile https://ethyca.atlassian.net/wiki/spaces/EN/pages/3029204995/2024-04-19+Google+Cloud+SQL+Integration#Option-2-(IAM-database-authentication)

galvana

This is a good first pass Andres! I was able to connect to a Google Cloud SQL for MySQL instance using keyfile creds and that was the main part of this ticket 👍 I think it's still good to test the retrieve_data and mask_data functions we're inheriting from SQLConnector so I documented some test files you can reference to do this.

I commented inline but I'll share here to make sure to run nox -s static_checks to identify any static issues like formatting, type, or import issues.

You will also run into some database migration issues once you merge in the main branch because your new migration and the latest migration both have the same down_revision

down_revision = "efddde14da21"

Search the repo for down_revision = "efddde14da21" and point your migration to the revision number of the other file with the same down_revision.

I know this is a lot so let me know if you want to go over this in person.

galvana · 2024-06-08T21:58:21Z

.github/workflows/backend_checks.yml

Thanks for sorting these values

noxfiles/setup_tests_nox.py

requirements.txt

galvana · 2024-06-08T22:06:30Z

src/fides/api/alembic/migrations/versions/98e0860a9320_add_google_cloud_sql_mysql_to_.py

+    # Add google_cloud_sql_mysql to ConnectionType enum
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb', 'bigquery', 'saas', 'manual', 'email', 'manual_webhook', 'timescale', 'fides', 'sovrn', 'google_cloud_sql_mysql')"


We're missing some connection types, make sure you include all the values in the ConnectionType enum

class ConnectionType(enum.Enum): """ Supported types to which we can connect Fides. """ postgres = "postgres" mongodb = "mongodb" mysql = "mysql" google_cloud_sql_mysql = "google_cloud_sql_mysql" https = "https" saas = "saas" redshift = "redshift" snowflake = "snowflake" mssql = "mssql" mariadb = "mariadb" bigquery = "bigquery" manual = "manual" # Deprecated - use manual_webhook instead sovrn = "sovrn" attentive = "attentive" dynamodb = "dynamodb" manual_webhook = "manual_webhook" # Runs upfront before the traversal timescale = "timescale" fides = "fides" generic_erasure_email = "generic_erasure_email" # Run after the traversal generic_consent_email = "generic_consent_email" # Run after the traversal

galvana · 2024-06-08T22:07:12Z

src/fides/api/alembic/migrations/versions/98e0860a9320_add_google_cloud_sql_mysql_to_.py

+    # Remove google_cloud_sql_mysql from the ConnectionType enum
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb', 'bigquery', 'saas', 'manual', 'email', 'manual_webhook', 'timescale', 'fides', 'sovrn')"


Same as above, make sure it's the same list as the upgrade but without the new google_cloud_sql_mysql entry

tests/ops/integration_tests/test_connection_configuration_integration.py

tests/ops/integration_tests/test_connection_google_sql_mysql_configuration_integration.py

src/fides/api/service/connectors/sql_connector.py

galvana · 2024-06-09T03:05:43Z

tests/fixtures/google_cloud_sql_mysql_fixtures.py

The connection test worked just fine which is the important part of this ticket! But we want to make sure that the access and erasure logic we're inheriting from the MySQLConnector still works for our new Google Cloud SQL instance.

To do. this, expand the fixtures to include something similar to mysql_integration_db from the mysql_fixtures.py file? This is what will create our test data in Google Cloud SQL.

@pytest.fixture(scope="function") def google_cloud_sql_mysql_integration_db(google_cloud_sql_mysql_integration_session): truncate_all_tables(mysql_integration_session) statements = [ """ INSERT INTO product VALUES (1, 'Example Product 1', 10.00), ...

Then add a test_integration_google_cloud_sql_mysql_example.py similar to tests/ops/integration_tests/test_integration_mysql_example.py to test that the data was created.

Then an equivalent of test_mysql_access_request_task in tests/ops/integration_tests/test_sql_task.py

And finally the most important part, a Google Cloud SQL version of test_create_and_process_access_request_mysql and test_create_and_process_erasure_request_specific_category_mysql in tests/ops/service/privacy_request/test_request_runner_service.py

If you search for @pytest.mark.integration_mysql in the repo, you should see all of these tests for the original version of the MySQL connector, make sure you have similar tests for your new connector. Add a new pytest marker in pyproject.toml for integration_google_cloud_sql_mysql so you can tag all of your new tests.

Co-authored-by: Adrian Galvan <[email protected]>

cypress · 2024-06-12T15:45:07Z

Passing run #8393 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `43bea8e` into `480312e`...
Project: fides	Commit: `d10f3a54c7 ℹ️`
Status: Passed	Duration: 00:35 💡
Started: Jun 19, 2024 1:03 AM	Ended: Jun 19, 2024 1:04 AM

Review all test suite changes for PR #4949 ↗︎

galvana · 2024-06-18T16:00:22Z

tests/ops/service/privacy_request/test_request_runner_service.py

+    "dsr_version",
+    ["use_dsr_3_0", "use_dsr_2_0"],
+)
+def test_create_and_process_erasure_request_specific_category_mysql(


Suggested change

def test_create_and_process_erasure_request_specific_category_mysql(

def test_create_and_process_erasure_request_specific_category_google_cloud_sql_mysql(

galvana

Almost there! Just a few minor things. Also make sure you run nox -s static_checks locally to format the files and pick up any other static issues.

src/fides/api/alembic/migrations/versions/7641ea685ea4_add_google_cloud_sql_mysql_to_.py

galvana · 2024-06-18T18:38:18Z

src/fides/api/schemas/connection_configuration/connection_secrets_google_cloud_sql_mysql.py

+
+    _required_components: List[str] = [
+        "db_iam_user",
+        "instance_connection_name",


Can you add dbname to _required_components now that it's required?

galvana · 2024-06-18T18:39:10Z

tests/fixtures/google_cloud_sql_mysql_fixtures.py

+    yield google_cloud_sql_mysql_integration_session
+
+
+# TODO: Consolidate these


Can we get rid of this TODO?

galvana · 2024-06-18T18:39:19Z

tests/fixtures/mysql_fixtures.py

@@ -95,6 +95,33 @@ def mysql_example_test_dataset_config(
    dataset.delete(db=db)
    ctl_dataset.delete(db=db)

+# TODO: Consolidate these


Same for this one

galvana · 2024-06-18T18:40:31Z

tests/fixtures/mysql_fixtures.py

@@ -95,6 +95,33 @@ def mysql_example_test_dataset_config(
    dataset.delete(db=db)
    ctl_dataset.delete(db=db)

+# TODO: Consolidate these
+@pytest.fixture
+def google_cloud_sql_mysql_example_test_dataset_config(


We can get rid of this one since we moved it to it's own file right?

data/dataset/google_cloud_sql_mysql_example_test_dataset.yml

Co-authored-by: Adrian Galvan <[email protected]>

noxfiles/setup_tests_nox.py

Co-authored-by: Adrian Galvan <[email protected]>

noxfiles/setup_tests_nox.py

data/dataset/google_cloud_sql_mysql_example_test_dataset.yml

codecov · 2024-06-19T01:31:00Z

Codecov Report

Attention: Patch coverage is 79.45205% with 15 lines in your changes missing coverage. Please review.

Project coverage is 86.56%. Comparing base (480312e) to head (43bea8e).

Files	Patch %	Lines
src/fides/api/service/connectors/sql_connector.py	54.54%	9 Missing and 1 partial ⚠️
...ation/connection_secrets_google_cloud_sql_mysql.py	90.00%	3 Missing ⚠️
src/fides/api/task/task_resources.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4949      +/-   ##
==========================================
- Coverage   86.59%   86.56%   -0.03%     
==========================================
  Files         351      352       +1     
  Lines       21818    21874      +56     
  Branches     2881     2884       +3     
==========================================
+ Hits        18893    18935      +42     
- Misses       2420     2433      +13     
- Partials      505      506       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

data/dataset/google_cloud_sql_mysql_example_test_dataset.yml

cypress · 2024-06-19T15:30:47Z

Passing run #8403 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Google Cloud SQL MySQL Connector (#4949)
Project: fides	Commit: `39d23e1ac6`
Status: Passed	Duration: 00:35 💡
Started: Jun 19, 2024 3:30 PM	Ended: Jun 19, 2024 3:30 PM

Review all test suite changes for PR #4949 ↗︎

andres-torres-marroquin added 5 commits May 24, 2024 12:30

Some steps forward integration Google Cloud SQL

7785f1e

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

4e59a66

Updates Google Cloud SQL MySQL Integration

d52fb51

Updates Google Cloud SQL MySQL Integration

e0d0101

Updates Google Cloud SQL MySQL Integration

40c0168

andres-torres-marroquin requested a review from galvana June 4, 2024 13:57

github-advanced-security bot found potential problems Jun 4, 2024

View reviewed changes

tests/ops/integration_tests/test_external_database_connections.py Fixed Show fixed Hide fixed

andres-torres-marroquin added 3 commits June 4, 2024 08:01

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

de71c40

Updates Google Cloud SQL MySQL Integration

b745c69

Updates Google Cloud SQL MySQL Integration

3f3fa9a

galvana reviewed Jun 4, 2024

View reviewed changes

Updates Google Cloud SQL MySQL Integration

4e71a1c

vercel bot deployed to Preview June 5, 2024 15:42 View deployment

andres-torres-marroquin added 2 commits June 6, 2024 11:19

Updates Google Cloud SQL MySQL Integration

0dc27df

Updates Google Cloud SQL MySQL Integration

6979b27

galvana requested changes Jun 9, 2024

View reviewed changes

andres-torres-marroquin and others added 4 commits June 10, 2024 07:51

Update noxfiles/setup_tests_nox.py

2109ed3

Co-authored-by: Adrian Galvan <[email protected]>

Update src/fides/api/service/connectors/sql_connector.py

f19cfe8

Co-authored-by: Adrian Galvan <[email protected]>

QA

60bd835

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

67fe4ea

vercel bot deployed to Preview June 10, 2024 19:02 View deployment

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

7de7c14

vercel bot deployed to Preview June 11, 2024 13:49 View deployment

andres-torres-marroquin added 5 commits June 11, 2024 11:28

QA

ab5c6a1

QA

54aa62f

QA

b7bdc8e

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

f856d1c

QA

a24cad9

galvana reviewed Jun 18, 2024

View reviewed changes

QA

286afac

andres-torres-marroquin changed the title ~~Google Cloud MySQL Integration Draft~~ Google Cloud SQL MySQL Connector Jun 18, 2024

Last bits of QA

f99cf3f

galvana self-requested a review June 18, 2024 18:33

galvana added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Jun 18, 2024

galvana reviewed Jun 18, 2024

View reviewed changes

andres-torres-marroquin and others added 2 commits June 18, 2024 12:48

Update data/dataset/google_cloud_sql_mysql_example_test_dataset.yml

b0392a3

Co-authored-by: Adrian Galvan <[email protected]>

Last bits of QA

e62fcab

galvana reviewed Jun 18, 2024

View reviewed changes

noxfiles/setup_tests_nox.py Outdated Show resolved Hide resolved

Update noxfiles/setup_tests_nox.py

5458b89

Co-authored-by: Adrian Galvan <[email protected]>

andres-torres-marroquin commented Jun 18, 2024

View reviewed changes

noxfiles/setup_tests_nox.py Show resolved Hide resolved

andres-torres-marroquin added 2 commits June 18, 2024 14:12

Update noxfiles/setup_tests_nox.py

58f188c

QA

ace74df

galvana reviewed Jun 18, 2024

View reviewed changes

data/dataset/google_cloud_sql_mysql_example_test_dataset.yml Outdated Show resolved Hide resolved

andres-torres-marroquin and others added 3 commits June 18, 2024 14:26

QA

4b0cce9

Typo fix

bfb908a

QA

fb5f3ca

galvana self-requested a review June 18, 2024 20:40

galvana added 5 commits June 18, 2024 15:29

Adding missing env variables for external tests

347dad0

Fixing test markers

659f0cc

Adding missing imports

00d36bf

Removing Google Cloud SQL tests from Safe-Tests and fixing dataset

8e87792

Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs

43bea8e

andres-torres-marroquin commented Jun 19, 2024

View reviewed changes

data/dataset/google_cloud_sql_mysql_example_test_dataset.yml Show resolved Hide resolved

galvana approved these changes Jun 19, 2024

View reviewed changes

andres-torres-marroquin merged commit 39d23e1 into main Jun 19, 2024
50 of 54 checks passed

andres-torres-marroquin deleted the PROD-2084-Google-Cloud-SQL-Integration-DSRs branch June 19, 2024 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google Cloud SQL MySQL Connector #4949

Google Cloud SQL MySQL Connector #4949

andres-torres-marroquin commented Jun 4, 2024 •

edited by jira bot

Loading

vercel bot commented Jun 4, 2024 •

edited

Loading

galvana Jun 4, 2024

galvana left a comment

galvana Jun 8, 2024

galvana Jun 8, 2024

galvana Jun 8, 2024

galvana Jun 9, 2024

cypress bot commented Jun 12, 2024 •

edited

Loading

galvana Jun 18, 2024

galvana left a comment

galvana Jun 18, 2024

galvana Jun 18, 2024

galvana Jun 18, 2024

galvana Jun 18, 2024

codecov bot commented Jun 19, 2024

cypress bot commented Jun 19, 2024

		)


		class GoogleCloudMySQLSchema(ConnectionConfigSecretsSchema):

	def test_create_and_process_erasure_request_specific_category_mysql(
	def test_create_and_process_erasure_request_specific_category_google_cloud_sql_mysql(

		yield google_cloud_sql_mysql_integration_session


		# TODO: Consolidate these

Google Cloud SQL MySQL Connector #4949

Google Cloud SQL MySQL Connector #4949

Conversation

andres-torres-marroquin commented Jun 4, 2024 • edited by jira bot Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

vercel bot commented Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

galvana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cypress bot commented Jun 12, 2024 • edited Loading

Passing run #8393 ↗︎

Review all test suite changes for PR #4949 ↗︎

Choose a reason for hiding this comment

galvana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 19, 2024

Codecov Report

cypress bot commented Jun 19, 2024

Passing run #8403 ↗︎

Review all test suite changes for PR #4949 ↗︎

andres-torres-marroquin commented Jun 4, 2024 •

edited by jira bot

Loading

vercel bot commented Jun 4, 2024 •

edited

Loading

cypress bot commented Jun 12, 2024 •

edited

Loading