Updating PreparererStep, creating PreparerMapping. CR changes to DataCleaner. #111

cchoquette · 2019-07-30T18:32:40Z

Description

Fixes #109

…adow internal workflow.

… DataPreparer; 3. DataCleaner functionality

…plementation testing its usage. Merge remote-tracking branch 'remotes/origin/development' into data_cleaner # Conflicts: # foreshadow/exceptions.py # foreshadow/transformers/base.py # pytest.ini

…ly implemented.

…names

…reparer

Merge in latest development.

Data Preparer and PreparerSteps base class and Data Cleaner

Merging in config from development. Merge branch 'development' into data_preparer # Conflicts: # foreshadow/core/serializers.py # foreshadow/tests/test_transformers/test_smart.py # pytest.ini

adithyabsk

A few more comments.

adithyabsk · 2019-07-31T20:46:18Z

foreshadow/tests/test_cleaners/test_data_cleaner.py

+    data = pd.DataFrame({"financials": np.arange(10)}, columns=columns)
+    cs = ColumnSharer()
+    dc = DataCleaner(column_sharer=cs)
+    transformed_data = dc.fit_transform(data)


Might you be able to test them in the same test if they require the same setup (I know I know it breaks my cardinal rule but creating a fixture isn't worth the overhead)

I'll look at it.

foreshadow/transformers/internals/__init__.py

…hanges. # Conflicts: # foreshadow/cleaners/data_cleaner.py # foreshadow/cleaners/internals/__init__.py # foreshadow/core/__init__.py # foreshadow/core/preparerstep.py # foreshadow/core/smarttransformer.py # foreshadow/core/wrapper.py # foreshadow/foreshadow.py # foreshadow/intents/registry.py # foreshadow/preprocessor.py # foreshadow/tests/test_transformers/test_transformers.py # foreshadow/transformers/concrete/externals.py # foreshadow/transformers/concrete/internals/__init__.py # foreshadow/transformers/core/pipeline.py # foreshadow/transformers/smart/smart.py

Included: tests skipped or changed. Some left failing to change as we integrate DataPreparer. V1 components removed V2 file structure in place with proper import system (some small changes still to be made).

adithyabsk

Added some comments lets chat.

foreshadow/console.py

foreshadow/foreshadow.py

foreshadow/metrics.py

foreshadow/optimizers/__init__.py

foreshadow/preparer/pipeline.py

foreshadow/concrete/internals/cleaners/__init__.py

foreshadow/concrete/internals/__init__.py

foreshadow/concrete/externals.py

foreshadow/concrete/__init__.py

adithyabsk

A few more comments.

adithyabsk · 2019-08-02T20:45:42Z

foreshadow/intents/__init__.py

@@ -0,0 +1,9 @@
+"""Intents package used by IntentMapper PreparerStep."""
+from foreshadow.intents.base import BaseIntent


Should this be relative

adithyabsk · 2019-08-02T20:46:31Z

foreshadow/old/__init__.py

@@ -0,0 +1 @@
+"""old stuff just kept for easier referencing."""


Make sure to ignore this folder in coverage

adithyabsk · 2019-08-02T20:49:51Z

foreshadow/preparer/preparer.py

@@ -2,9 +2,9 @@

 from sklearn.pipeline import Pipeline

+from foreshadow.intents import IntentMapper


shouldn't this be in steps

adithyabsk · 2019-08-02T20:51:07Z

foreshadow/preparer/preparerstep.py

@@ -137,9 +137,12 @@ def _check_parallelizable_batch(group_process, group_number):
    """
    if group_process.single_input is not None:
        inputs = group_process.single_input
-        steps = [


Add the None option to the docs

adithyabsk · 2019-08-02T20:56:20Z

foreshadow/tests/test_core/test_resolver.py

@@ -7,12 +7,12 @@ def test_resolver_overall():
    import numpy as np
    import pandas as pd
    from foreshadow.preparer.column_sharer import ColumnSharer
-    from foreshadow.preparer.steps.resolver import ResolverMapper
+    from foreshadow.preparer.steps.mapper import IntentMapper


This just needs to be imported from preparer.steps

adithyabsk · 2019-08-02T20:58:18Z

foreshadow/tests/test_transformers/test_concrete/test_cleaners/test_data_cleaner.py

@@ -44,6 +44,8 @@ def test_data_cleaner_fit():
            "financials",
        ],
    )
+    print(data.values)
+    print(check.values)


Remove these prints

adithyabsk · 2019-08-02T20:58:27Z

foreshadow/tests/test_transformers/test_concrete/test_cleaners/test_data_cleaner.py

-    from foreshadow.preparer.steps import CleanerMapper
-    from foreshadow.preparer.column_sharer import ColumnSharer
+    from foreshadow.preparer import CleanerMapper
+    from foreshadow.preparer import ColumnSharer


adithyabsk · 2019-08-02T20:58:36Z

foreshadow/tests/test_transformers/test_concrete/test_cleaners/test_data_cleaner.py

-    from foreshadow.preparer.steps import CleanerMapper
-    from foreshadow.preparer.column_sharer import ColumnSharer
+    from foreshadow.preparer import CleanerMapper
+    from foreshadow.preparer import ColumnSharer


This entire file

adithyabsk · 2019-08-02T20:59:01Z

foreshadow/tests/test_transformers/test_smart/test_smart.py

-    from foreshadow.concrete.internals import FixedTfidfVectorizer
-    from foreshadow.concrete.internals import HTMLRemover
+    from foreshadow.concrete import FixedTfidfVectorizer
+    from foreshadow.concrete import HTMLRemover


adithyabsk · 2019-08-02T20:59:35Z

foreshadow/utils/common.py

    """Get the cache path which is in the config directory.

    Note:
        This function also makes the directory if it does not already exist.

-    Args:
-        path (str): A path to override the cache save directory path.
-
    Returns:
        str; The path to the cache directory.


This should be a colon not semi-colon

cchoquette and others added 30 commits July 11, 2019 13:16

Outline for DataPreparer object as defined pipeline object for Foresh…

711ccea

…adow internal workflow.

Extended 1.DataPreparer functionality; 2. Base Class for each step of…

2647236

… DataPreparer; 3. DataCleaner functionality

Fixing pytest.ini file for command line usage.

7413c7f

Removing modeler from data_preparer

d59e4bb

Switching _none_to_dict to single kwarg

2625e87

Making unprivate imports

5c2033e

Fixing __init__'s

58fdb8b

Adding financial_cleaner example

80bd55a

Committing prior to merging in development.

75decdf

New PreparerStep base class. Minimal tests and minimal DataCleaner im…

cd13449

…plementation testing its usage. Merge remote-tracking branch 'remotes/origin/development' into data_cleaner # Conflicts: # foreshadow/exceptions.py # foreshadow/transformers/base.py # pytest.ini

Cleaned up PreparerStep base class code and added thorough doc strings.

dd910b2

Fixing pytest.ini

ed30660

DataCleaner working without full tests less creating new columns

c7d14d0

Finished DataPreparer and DataCleaner without tests/DropTransform ful…

d64a524

…ly implemented.

Adding test and fixing DataCleaner pipeline creation to include step …

5f78dd9

…names

Merge remote-tracking branch 'remotes/origin/development' into data_p…

ef97754

…reparer

merge in data_preparer

a67d197

Merge branch 'data_preparer' into data_cleaner

2e67a71

Merge branch 'development' into data_preparer

a3e81d4

Almost fully working solution.

d4bfeae

Merge in latest development.

DataCleaner working for a DataFrame with 1 column.

9b5018d

Working PreparerSteps and DataCleaner, with a couple tests to show.

ef73579

Making smart still fit aggregate transformer, but return self.

b2e409a

Changed column naming scheme and added column_sharer across the project.

9576014

Fully working solution with DropTransform. Minimal Tests.

85633dc

Flaked.

a8f85ab

isorted.

de2b0ff

Removed DropMixin

0971f4c

Data Preparer and PreparerSteps base class and Data Cleaner(#98)

b0cda87

Data Preparer and PreparerSteps base class and Data Cleaner

DataPreparer, Baseclass for PreparerSteps, DataCleaner.

02cbf32

Merging in config from development. Merge branch 'development' into data_preparer # Conflicts: # foreshadow/core/serializers.py # foreshadow/tests/test_transformers/test_smart.py # pytest.ini

adithyabsk reviewed Jul 31, 2019

View reviewed changes

cchoquette added 10 commits August 1, 2019 07:22

Project restructure.

629d940

Project restructure.

b644abc

Partial project restructure.

5027969

Partial project restructure.

84b9516

Final Project restructure:

a8550b3

Included: tests skipped or changed. Some left failing to change as we integrate DataPreparer. V1 components removed V2 file structure in place with proper import system (some small changes still to be made).

foreshadow.concrete import rollup complete.

7dd156c

Partial flake

9bca84b

Flake complete.

016e7a0

Fixing setup.cfg

3bddf65

adithyabsk suggested changes Aug 2, 2019

View reviewed changes

This was referenced Aug 2, 2019

Look at if \_\_all\_\_ is needed in metrics.py #121

Open

Fix logging imports. #122

Closed

Move V1 kwargs to ColumnSharer #123

Closed

Fix to config. #124

Closed

Fix foreshadow object. #125

Closed

Make ColumnSharer Thread Safe #127

Closed

cchoquette added 6 commits August 2, 2019 16:14

Code review changes.

81b6fbf

Flaked.

3133b1f

More flake.

a96f841

Skipping 5 important tests for next sprint.

7c27697

skipping integration.

34ebff0

skipping integration.

0afce1e

adithyabsk approved these changes Aug 2, 2019

View reviewed changes

cchoquette merged commit c8cd26b into development Aug 2, 2019

adithyabsk deleted the data_preparer branch August 6, 2019 19:28

adithyabsk mentioned this pull request Aug 6, 2019

Create DataCleaner Class and Implementation #94

Closed

This was referenced Aug 7, 2019

Make ColumnSharer and Config Process Safe (low priority) #130

Closed

Switch patching to use patchy #109

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating PreparererStep, creating PreparerMapping. CR changes to DataCleaner. #111

Updating PreparererStep, creating PreparerMapping. CR changes to DataCleaner. #111

cchoquette commented Jul 30, 2019 •

edited

Loading

adithyabsk left a comment

adithyabsk Jul 31, 2019

cchoquette Aug 2, 2019

adithyabsk left a comment

adithyabsk left a comment

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

adithyabsk Aug 2, 2019

		@@ -0,0 +1,9 @@
		"""Intents package used by IntentMapper PreparerStep."""
		from foreshadow.intents.base import BaseIntent

		@@ -0,0 +1 @@
		"""old stuff just kept for easier referencing."""

		@@ -2,9 +2,9 @@

		from sklearn.pipeline import Pipeline

		from foreshadow.intents import IntentMapper

Updating PreparererStep, creating PreparerMapping. CR changes to DataCleaner. #111

Updating PreparererStep, creating PreparerMapping. CR changes to DataCleaner. #111

Conversation

cchoquette commented Jul 30, 2019 • edited Loading

Description

adithyabsk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adithyabsk left a comment

Choose a reason for hiding this comment

adithyabsk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cchoquette commented Jul 30, 2019 •

edited

Loading