Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google Cloud SQL MySQL Connector #4949

Merged
Merged
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
7785f1e
Some steps forward integration Google Cloud SQL
andres-torres-marroquin May 24, 2024
4e59a66
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 3, 2024
d52fb51
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 3, 2024
e0d0101
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 4, 2024
40c0168
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 4, 2024
de71c40
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 4, 2024
b745c69
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 4, 2024
3f3fa9a
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 4, 2024
4e71a1c
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 5, 2024
0dc27df
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 6, 2024
6979b27
Updates Google Cloud SQL MySQL Integration
andres-torres-marroquin Jun 6, 2024
2109ed3
Update noxfiles/setup_tests_nox.py
andres-torres-marroquin Jun 10, 2024
f19cfe8
Update src/fides/api/service/connectors/sql_connector.py
andres-torres-marroquin Jun 10, 2024
60bd835
QA
andres-torres-marroquin Jun 10, 2024
67fe4ea
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 10, 2024
7de7c14
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 11, 2024
ab5c6a1
QA
andres-torres-marroquin Jun 11, 2024
54aa62f
QA
andres-torres-marroquin Jun 11, 2024
b7bdc8e
QA
andres-torres-marroquin Jun 12, 2024
f856d1c
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 12, 2024
a24cad9
QA
andres-torres-marroquin Jun 12, 2024
73530c8
QA
andres-torres-marroquin Jun 12, 2024
77a8b4f
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 12, 2024
6c16fdc
QA
andres-torres-marroquin Jun 12, 2024
3c0e61b
QA
andres-torres-marroquin Jun 12, 2024
ca5943a
QA
andres-torres-marroquin Jun 13, 2024
04110cc
QA
andres-torres-marroquin Jun 13, 2024
0fb4f91
QA
andres-torres-marroquin Jun 13, 2024
09c0a7a
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 13, 2024
1c2a3c9
QA
andres-torres-marroquin Jun 14, 2024
cb46034
QA
andres-torres-marroquin Jun 17, 2024
7f3bc61
QA
andres-torres-marroquin Jun 17, 2024
da682bf
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
galvana Jun 17, 2024
b59ee98
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
galvana Jun 17, 2024
e6ba1a3
Adding icon for Google Cloud SQL
galvana Jun 17, 2024
4a5f3d4
Adding last test for GCP on the request runner
andres-torres-marroquin Jun 18, 2024
a9fec2a
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
andres-torres-marroquin Jun 18, 2024
286afac
QA
andres-torres-marroquin Jun 18, 2024
f99cf3f
Last bits of QA
andres-torres-marroquin Jun 18, 2024
b0392a3
Update data/dataset/google_cloud_sql_mysql_example_test_dataset.yml
andres-torres-marroquin Jun 18, 2024
e62fcab
Last bits of QA
andres-torres-marroquin Jun 18, 2024
5458b89
Update noxfiles/setup_tests_nox.py
andres-torres-marroquin Jun 18, 2024
58f188c
Update noxfiles/setup_tests_nox.py
andres-torres-marroquin Jun 18, 2024
ace74df
QA
andres-torres-marroquin Jun 18, 2024
4b0cce9
QA
andres-torres-marroquin Jun 18, 2024
bfb908a
Typo fix
galvana Jun 18, 2024
fb5f3ca
QA
andres-torres-marroquin Jun 18, 2024
347dad0
Adding missing env variables for external tests
galvana Jun 18, 2024
659f0cc
Fixing test markers
galvana Jun 18, 2024
00d36bf
Adding missing imports
galvana Jun 19, 2024
8e87792
Removing Google Cloud SQL tests from Safe-Tests and fixing dataset
galvana Jun 19, 2024
43bea8e
Merge branch 'main' into PROD-2084-Google-Cloud-SQL-Integration-DSRs
galvana Jun 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions .github/workflows/backend_checks.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sorting these values

Original file line number Diff line number Diff line change
Expand Up @@ -337,23 +337,27 @@ jobs:

- name: Integration Tests (External)
env:
BIGQUERY_KEYFILE_CREDS: ${{ secrets.BIGQUERY_KEYFILE_CREDS }}
BIGQUERY_DATASET: fidesopstest
DYNAMODB_REGION: ${{ secrets.DYNAMODB_REGION }}
BIGQUERY_KEYFILE_CREDS: ${{ secrets.BIGQUERY_KEYFILE_CREDS }}
DYNAMODB_ACCESS_KEY_ID: ${{ secrets.DYNAMODB_ACCESS_KEY_ID }}
DYNAMODB_ACCESS_KEY: ${{ secrets.DYNAMODB_ACCESS_KEY }}
DYNAMODB_REGION: ${{ secrets.DYNAMODB_REGION }}
GOOGLE_CLOUD_SQL_MYSQL_DB_IAM_USER: ${{ secrets.GOOGLE_CLOUD_SQL_MYSQL_DB_IAM_USER }}
GOOGLE_CLOUD_SQL_MYSQL_INSTANCE_CONNECTION_NAME: ${{ secrets.GOOGLE_CLOUD_SQL_MYSQL_INSTANCE_CONNECTION_NAME }}
GOOGLE_CLOUD_SQL_MYSQL_DATABASE_NAME: ${{ secrets.GOOGLE_CLOUD_SQL_MYSQL_DATABASE_NAME }}
GOOGLE_CLOUD_SQL_MYSQL_KEYFILE_CREDS: ${{ secrets.GOOGLE_CLOUD_SQL_MYSQL_KEYFILE_CREDS }}
REDSHIFT_TEST_DATABASE: ${{ secrets.REDSHIFT_TEST_DATABASE }}
REDSHIFT_TEST_DB_SCHEMA: ${{ secrets.REDSHIFT_TEST_DB_SCHEMA }}
REDSHIFT_TEST_HOST: ${{ secrets.REDSHIFT_TEST_HOST }}
REDSHIFT_TEST_PASSWORD: ${{ secrets.REDSHIFT_TEST_PASSWORD }}
REDSHIFT_TEST_PORT: ${{ secrets.REDSHIFT_TEST_PORT }}
REDSHIFT_TEST_USER: ${{ secrets.REDSHIFT_TEST_USER }}
REDSHIFT_TEST_PASSWORD: ${{ secrets.REDSHIFT_TEST_PASSWORD }}
REDSHIFT_TEST_DATABASE: ${{ secrets.REDSHIFT_TEST_DATABASE }}
REDSHIFT_TEST_DB_SCHEMA: ${{ secrets.REDSHIFT_TEST_DB_SCHEMA }}
SNOWFLAKE_TEST_ACCOUNT_IDENTIFIER: ${{ secrets.SNOWFLAKE_TEST_ACCOUNT_IDENTIFIER }}
SNOWFLAKE_TEST_USER_LOGIN_NAME: ${{ secrets.SNOWFLAKE_TEST_USER_LOGIN_NAME }}
SNOWFLAKE_TEST_PASSWORD: ${{ secrets.SNOWFLAKE_TEST_PASSWORD }}
SNOWFLAKE_TEST_WAREHOUSE_NAME: ${{ secrets.SNOWFLAKE_TEST_WAREHOUSE_NAME }}
SNOWFLAKE_TEST_DATABASE_NAME: ${{ secrets.SNOWFLAKE_TEST_DATABASE_NAME }}
SNOWFLAKE_TEST_PASSWORD: ${{ secrets.SNOWFLAKE_TEST_PASSWORD }}
SNOWFLAKE_TEST_SCHEMA_NAME: ${{ secrets.SNOWFLAKE_TEST_SCHEMA_NAME }}
SNOWFLAKE_TEST_USER_LOGIN_NAME: ${{ secrets.SNOWFLAKE_TEST_USER_LOGIN_NAME }}
SNOWFLAKE_TEST_WAREHOUSE_NAME: ${{ secrets.SNOWFLAKE_TEST_WAREHOUSE_NAME }}

run: nox -s "pytest(ops-external-datastores)"

Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ The types of changes are:
- Support for Limited FIDES__CELERY__* Env Vars [#4980](https://github.com/ethyca/fides/pull/4980)
- Implement sending emails via property-specific messaging templates [#4950](https://github.com/ethyca/fides/pull/4950)
- New privacy request search to replace existing endpoint [#4987](https://github.com/ethyca/fides/pull/4987)
- Added new Google Cloud SQL for MySQL Connector [#4949](https://github.com/ethyca/fides/pull/4949)

### Changed
- Move new data map reporting table out of beta and remove old table from Data Lineage map. [#4963](https://github.com/ethyca/fides/pull/4963)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@
"human_readable": "MySQL",
"encoded_icon": null
},
{
"identifier": "google_cloud_sql_mysql",
"type": "database",
"human_readable": "Google Cloud SQL for MySQL",
"encoded_icon": null
},
{
"identifier": "postgres",
"type": "database",
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ export const CONNECTION_TYPE_LOGO_MAP = new Map<ConnectionType, string>([
[ConnectionType.DYNAMODB, "dynamodb.svg"],
[ConnectionType.GENERIC_CONSENT_EMAIL, "ethyca.svg"],
[ConnectionType.GENERIC_ERASURE_EMAIL, "ethyca.svg"],
[ConnectionType.GOOGLE_CLOUD_SQL_MYSQL, "google_cloud_sql.svg"],
[ConnectionType.MANUAL_WEBHOOK, "manual_webhook.svg"],
[ConnectionType.MARIADB, "mariadb.svg"],
[ConnectionType.MONGODB, "mongodb.svg"],
Expand Down
1 change: 1 addition & 0 deletions clients/admin-ui/src/types/api/models/ConnectionType.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@ export enum ConnectionType {
SCYLLA = "scylla",
GENERIC_ERASURE_EMAIL = "generic_erasure_email",
GENERIC_CONSENT_EMAIL = "generic_consent_email",
GOOGLE_CLOUD_SQL_MYSQL = "google_cloud_sql_mysql",
}
210 changes: 210 additions & 0 deletions data/dataset/google_cloud_sql_mysql_example_test_dataset.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
dataset:
- fides_key: mysql_example_test_dataset
name: MySQL Example Test Dataset
description: Example of a MySQL dataset containing a variety of related tables like customers, products, addresses, etc.
andres-torres-marroquin marked this conversation as resolved.
Show resolved Hide resolved
collections:
- name: address
fields:
- name: city
data_categories: [user.contact.address.city]
- name: house
data_categories: [user.contact.address.street]
- name: id
data_categories: [system.operations]
fides_meta:
primary_key: True
- name: state
data_categories: [user.contact.address.state]
- name: street
data_categories: [user.contact.address.street]
- name: zip
data_categories: [user.contact.address.postal_code]

- name: customer
fields:
- name: address_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: address.id
direction: to
- name: created
data_categories: [system.operations]
- name: email
data_categories: [user.contact.email]
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [user.unique_id]
fides_meta:
primary_key: True
- name: name
data_categories: [user.name]

- name: employee
fields:
- name: address_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: address.id
direction: to
- name: email
data_categories: [user.contact.email]
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [user.unique_id]
fides_meta:
primary_key: True
- name: name
data_categories: [user.name]

- name: login
fields:
- name: customer_id
data_categories: [user.unique_id]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
- name: time
data_categories: [user.sensor]

- name: orders
fields:
- name: customer_id
data_categories: [user.unique_id]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
fides_meta:
primary_key: True
- name: shipping_address_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: address.id
direction: to

# order_item
- name: order_item
fields:
- name: order_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: orders.id
direction: from
- name: product_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: product.id
direction: to
- name: quantity
data_categories: [system.operations]

- name: payment_card
fields:
- name: billing_address_id
data_categories: [system.operations]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: address.id
direction: to
- name: ccn
data_categories: [user.financial.bank_account]
- name: code
data_categories: [user.financial]
- name: customer_id
data_categories: [user.unique_id]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: customer.id
direction: from
- name: id
data_categories: [system.operations]
- name: name
data_categories: [user.financial]
- name: preferred
data_categories: [user]

- name: product
fields:
- name: id
data_categories: [system.operations]
- name: name
data_categories: [system.operations]
- name: price
data_categories: [system.operations]

- name: report
fields:
- name: email
data_categories: [user.contact.email]
fides_meta:
identity: email
data_type: string
- name: id
data_categories: [system.operations]
- name: month
data_categories: [system.operations]
- name: name
data_categories: [system.operations]
- name: total_visits
data_categories: [system.operations]
- name: year
data_categories: [system.operations]

- name: service_request
fields:
- name: alt_email
data_categories: [user.contact.email]
fides_meta:
identity: email
data_type: string
- name: closed
data_categories: [system.operations]
- name: email
data_categories: [system.operations]
fides_meta:
identity: email
data_type: string
- name: employee_id
data_categories: [user.unique_id]
fides_meta:
references:
- dataset: mysql_example_test_dataset
field: employee.id
direction: from
- name: id
data_categories: [system.operations]
- name: opened
data_categories: [system.operations]

- name: visit
fields:
- name: email
data_categories: [user.contact.email]
fides_meta:
identity: email
data_type: string
- name: last_visit
data_categories: [system.operations]
6 changes: 6 additions & 0 deletions noxfiles/setup_tests_nox.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,12 @@ def pytest_ops(session: Session, mark: str, coverage_arg: str) -> None:
"-e",
"BIGQUERY_DATASET",
"-e",
"-e",
andres-torres-marroquin marked this conversation as resolved.
Show resolved Hide resolved
"GOOGLE_CLOUD_SQL_MYSQL_DB_IAM_USER",
"-e",
"GOOGLE_CLOUD_SQL_MYSQL_INSTANCE_CONNECTION_NAME",
"-e",
"GOOGLE_CLOUD_SQL_MYSQL_KEYFILE_CREDS",
andres-torres-marroquin marked this conversation as resolved.
Show resolved Hide resolved
"DYNAMODB_REGION",
"-e",
"DYNAMODB_ACCESS_KEY_ID",
Expand Down
8 changes: 5 additions & 3 deletions pyproject.toml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add integration_google_cloud_sql_mysql to the markers value in this file? And make sure you add the marker to all of your new Google Cloud SQL for MySQL tests?

@pytest.mark.integration_google_cloud_sql_mysql
...

Original file line number Diff line number Diff line change
Expand Up @@ -47,27 +47,28 @@ module = [
"fideslog.*",
"firebase_admin.*",
"google.api_core.*",
"nh3.*",
"joblib.*",
"jose.*",
"jwt.*",
"multidimensional_urlencode.*",
"networkx.*",
"nh3.*",
"okta.*",
"pandas.*",
"plotly.*",
"pydash.*",
"pygtrie.*",
"pymongo.*",
"pymysql.*",
"RestrictedPython.*",
"sendgrid.*",
"snowflake.*",
"sqlalchemy_utils.*",
"sqlalchemy.ext.*",
"sqlalchemy.future.*",
"sqlalchemy_utils.*",
"twilio.*",
"uvicorn.*",
"validators.*",
"pygtrie.*",
]
ignore_missing_imports = true

Expand Down Expand Up @@ -185,6 +186,7 @@ markers = [
"integration_postgres",
"integration_mongodb",
"integration_mssql",
"integration_google_cloud_sql_mysql",
"integration_redshift",
"integration_snowflake",
"integration_mariadb",
Expand Down
3 changes: 2 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ APScheduler==3.9.1.post1
asyncpg==0.27.0
boto3==1.26.1
celery[pytest]==5.2.7
cloud-sql-python-connector==1.9.2
colorama>=0.4.3
cryptography==38.0.3
cryptography==42.0.0
galvana marked this conversation as resolved.
Show resolved Hide resolved
dask==2022.9.2
deepdiff==6.3.0
defusedxml==0.7.1
Expand Down
Loading
Loading