Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ETL-409] Create python comparison script #42

Merged
merged 19 commits into from
Apr 27, 2023
Merged

[ETL-409] Create python comparison script #42

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
33b1255
add initial compare parquet datasets pythob job code, and tests
Apr 14, 2023
7e9ed35
add docstring and comments
Apr 14, 2023
393cb19
update code to use datacompy as main comparison tool
Apr 14, 2023
4174e0c
remove all unused functions and tests covered by datacompy, fix metho…
Apr 15, 2023
da6dafc
add syntax updates, clean up parquet dataset test, add comparison rep…
Apr 17, 2023
c00090b
add tests for compare_datasets_by_data_type, add a small validation f…
Apr 17, 2023
6f9415e
remove debugging, add test for checking has_common_cols logic in comp…
Apr 17, 2023
1daffef
add os.path.join for more robustness
Apr 18, 2023
91e646e
add additional printouts of entire missing rows, add rows, mismatch c…
Apr 20, 2023
a51baee
add function for converting csv to text for s3.put_object, add tests …
Apr 20, 2023
ba12476
add duplicates report, add TODOs, add more robust tests, re-org savin…
Apr 20, 2023
a3ef38b
add more TODOs
Apr 20, 2023
52dcca2
use resolvedoptions from aws glue, adjust all args and args validatio…
Apr 21, 2023
81dfc82
Merge branch 'main' into etl-409
Apr 21, 2023
4c96216
add datatype subtypes handling
Apr 24, 2023
c2cc21f
add exception catching by dataset
Apr 24, 2023
3477c18
add conversion to categorical, make it run per data type
Apr 26, 2023
2888fe9
add logging for memory for datasets
Apr 27, 2023
c6a4184
remove categorical conversion, fix test name
Apr 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ pre-commit = "*"
sceptre = ">=3.2.0"
sceptre-sam-handler = "*"
synapseclient = "~=2.7"
pandas = "<1.5"
moto = "~=4.1"
datacompy = "~=0.8"
Loading