[ETL-409] Create python comparison script (#42)

* update code to use datacompy as main comparison tool * add comparison report for edge scenarios like no data types in common, dups * add function for converting csv to text for s3.put_object, add tests for getting all row diff, add linting, adjust function to output datacompy.Compare object to allow for more logs * use resolvedoptions from aws glue, adjust all args and args validation func, remove unused lib * add logging for memory for datasets --------- Co-authored-by: Rixing Xu <[email protected]>
Sage-Bionetworks · Apr 27, 2023 · 0ab9b99 · 0ab9b99
1 parent 754fe3d
commit 0ab9b99
Show file tree

Hide file tree

Showing 5 changed files with 1,897 additions and 234 deletions.
diff --git a/Pipfile b/Pipfile
@@ -13,3 +13,6 @@ pre-commit = "*"
 sceptre = ">=3.2.0"
 sceptre-sam-handler = "*"
 synapseclient = "~=2.7"
+pandas = "<1.5"
+moto = "~=4.1"
+datacompy = "~=0.8"