Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bug where fillna assumes each column is its own partition #51

Merged
merged 2 commits into from
Jul 20, 2018
Merged

Fix bug where fillna assumes each column is its own partition #51

merged 2 commits into from
Jul 20, 2018

Conversation

devin-petersohn
Copy link
Collaborator

@devin-petersohn devin-petersohn commented Jul 20, 2018 •
edited
Loading

What do these changes do?

Fixes the bug where each column/row is assumed to be unique on the partition.

Related issue number

#50

  • passes git diff upstream/master -u -- "*.py" | flake8 --diff

@kunalgosar kunalgosar added the bug 🦗 Something isn't working label Jul 20, 2018
simon-mo
simon-mo approved these changes Jul 20, 2018
View reviewed changes
Copy link
Collaborator

@simon-mo simon-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I believe this issue is general. Many of our implementation assumed fine-grained partition without considering the case for efficiently mutating large block dataframe

@simon-mo simon-mo merged commit 9826ea7 into modin-project:master Jul 20, 2018
@simon-mo simon-mo mentioned this pull request Jul 20, 2018
Closed
dchigarev pushed a commit to dchigarev/modin that referenced this pull request Aug 25, 2020
@ienkovich
Merge pull request modin-project#51 from intel-go/ienkovich/remove-ex…
e368399 
…ec-triggers

Ienkovich/remove exec triggers
pyrito pushed a commit to pyrito/modin that referenced this pull request Feb 3, 2023
@helmeleegy
Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpa…
f7a31ab 
…rtition (modin-project#51)

* Fix dt_day_of_week/day_of_year, str_partition/replace/rpartition

* Fix str_extract
pyrito pushed a commit to pyrito/modin that referenced this pull request Feb 3, 2023
@helmeleegy
Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/partition/rep…
ad9231d 
…lace/rpartition (modin-project#51)" (modin-project#58)

This reverts commit f7a31ab.
pyrito pushed a commit to pyrito/modin that referenced this pull request Feb 3, 2023
@helmeleegy
Revert "Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/parti…
42845f6 
…tion/replace/rpartition (modin-project#51)" (modin-project#58)" (modin-project#60)

This reverts commit ad9231d.
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Feb 13, 2023
@helmeleegy @vnlitvinov
Fix str_cat/extract/partition/replace/rpartition (modin-project#51) (…
35797e9 
…core)
RehanSD pushed a commit to RehanSD/modin that referenced this pull request Feb 17, 2023
@devin-petersohn
FEAT-modin-project#4931: Create a query compiler that can connect to …
20653a4 
…a service

Signed-off-by: Devin Petersohn <[email protected]>

Fixes to pass CI + docs for io.py

Update implementation

Signed-off-by: Devin Petersohn <[email protected]>

Fix some things

Signed-off-by: Devin Petersohn <[email protected]>

Lint fixes

Fix put

Signed-off-by: Devin Petersohn <[email protected]>

Clean up and add new details

Signed-off-by: Devin Petersohn <[email protected]>

Use fsspec to get full path and allow URLs

Signed-off-by: Devin Petersohn <[email protected]>

Add lazy loc

Signed-off-by: Devin Petersohn <[email protected]>

fixes for tests

porting more tests

more fixes

moar fixes

Raise exception

Signed-off-by: Devin Petersohn <[email protected]>

Lint fixes

Return Python as the default modin engine

Handle indexing case for client qc

Call fast path for __getitem__ if not lazy

Remove user warning for Python-engine fall back

Add init

Signed-off-by: Devin Petersohn <[email protected]>

Implement free as a no-op

Signed-off-by: Devin Petersohn <[email protected]>

Add support for replace - client side

Fix a couple of issues with Client

Signed-off-by: Devin Petersohn <[email protected]>

Throw errors on to_pandas

Signed-off-by: Devin Petersohn <[email protected]>

Do not default to pandas for str_repeat

Add support for 18 datetime functions/properties

Fix columns caching when renaming columns

Fix test_query: put backticks back for col names

Add support for astype -- client side

hard coded changes for functions

Client support for str_(en/de)code, to_datetime

Add all missing query compiler methods.

Signed-off-by: mvashishtha <[email protected]>

Fix getitem_column_array and take_2d.

Signed-off-by: mvashishtha <[email protected]>

Fix getitem_column_array and take_2d.

Signed-off-by: mvashishtha <[email protected]>

Fix again.

Signed-off-by: mvashishtha <[email protected]>

Fix more bugs.

Signed-off-by: mvashishtha <[email protected]>

More fixes.

Signed-off-by: mvashishtha <[email protected]>

Fix more bugs-- pushdown tests test_dates and test_pivot still broken due to service bugs.

Signed-off-by: mvashishtha <[email protected]>

Fix typo. Note drop() broken because service requires you to specify both argument and client QC at base of this PR uses default Nones.

Signed-off-by: mvashishtha <[email protected]>

Add query compiler class.

Signed-off-by: mvashishtha <[email protected]>

Testing a commit

Initial changes for adding support for Expanding

FEAT Support for rolling.sem

FEAT support for Expanding sum, min, max, mean, var, std, count, sem

Removing extratenous comment

REFACTOR: Remove defaults to pandas at API layer and add some corresponding client QC methods.

Signed-off-by: mvashishtha <[email protected]>

Add more methods.

Signed-off-by: mvashishtha <[email protected]>

Fix expanding.

Signed-off-by: mvashishtha <[email protected]>

Add ewm.

Signed-off-by: mvashishtha <[email protected]>

Revert whitespace.

Signed-off-by: mvashishtha <[email protected]>

Fix to_numpy by making it like to_pandas.

Signed-off-by: mvashishtha <[email protected]>

Remove extra to_numpy.

Signed-off-by: mvashishtha <[email protected]>

Pass kwargs

Signed-off-by: mvashishtha <[email protected]>

Fix DataFrame import for isin.

Signed-off-by: mvashishtha <[email protected]>

Fix again.

Signed-off-by: mvashishtha <[email protected]>

Remove breakpoint

Signed-off-by: mvashishtha <[email protected]>

Tell if series.

Signed-off-by: mvashishtha <[email protected]>

Fix client qc.

Signed-off-by: mvashishtha <[email protected]>

Add self_is_series.

Signed-off-by: mvashishtha <[email protected]>

FIX: Set numeric_only to True in groupby quantile

Add some comments

Fix str_cat/fullmatch/removeprefix/removesuffix/translate/wrap (modin-project#44)

* Fix str_cat/fullmatch/removeprefix/removesuffix/translate/wrap

* Update modin/core/storage_formats/base/query_compiler.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

* Update modin/pandas/series_utils.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

* Update modin/core/storage_formats/base/query_compiler.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

Co-authored-by: Mahesh Vashishtha <[email protected]>

FEAT Support expanding.aggregate (modin-project#45)

Fix at_time and between_time. (modin-project#43)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

Add QC method for groupby.sem (modin-project#47)

* FEAT: Add partial support for groupby.sem()

* Add sem changes to groupby

Fix nlargest and nsmallest Series support (modin-project#46)

* Fix nlargest and smallest support

Signed-off-by: Naren Krishna <[email protected]>

Remove client query compiler's columnarize. (modin-project#48)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

Fix info and set memory_usage=False. (modin-project#49)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

POND-815 fixes for 21 column dataset (modin-project#50)

* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>

Bring in upstream series binary operation fix 6d5545f… (modin-project#52)

* Bring in upstream series binary operation fix 6d5545f.

Signed-off-by: mvashishtha <[email protected]>

* Update modin/pandas/series.py

Co-authored-by: Karthik Velayutham <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>
Co-authored-by: Karthik Velayutham <[email protected]>

Support groupby first/last (modin-project#53)

Signed-off-by: Naren Krishna <[email protected]>

FEAT: Add initial partial support for groupby.cumcount() (modin-project#54)

* FEAT: Add partial support for cumcount

* Remove the set_index_name

* Squeeze the result

* Write cumcount name to None

* Can't set dtype to int64

Fix resample sum, prod, size (modin-project#56)

Signed-off-by: Naren Krishna <[email protected]>

POND-184: fix describe and simplify query compiler interface (modin-project#55)

* Fix describe

Signed-off-by: mvashishtha <[email protected]>

* Pass datetime_is_numeric.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)

* Fix dt_day_of_week/day_of_year, str_partition/replace/rpartition

* Fix str_extract

Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)" (modin-project#58)

This reverts commit f7a31ab.

Revert "Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)" (modin-project#58)" (modin-project#60)

This reverts commit ad9231d.

Add query compiler method for groupby.prod() (modin-project#57)

Signed-off-by: Naren Krishna <[email protected]>

FEAT: Add support for groupby.head and groupby.tail (modin-project#61)

* FEAT: Add support for groupby.head and groupby.tail

* Change _change_index

FEAT: Add partial support for groupby.nth (modin-project#62)

FIX: Push first and last down to query compiler. (modin-project#64)

* FIX: Push first and last down to query compiler.

Signed-off-by: mvashishtha <[email protected]>

* Fix last.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FEAT: Add partial support for groupby.ngroup (modin-project#65)

* FEAT: Add partial support for groupby.ngroup

* Name of result should be none for now

Add client support for SeriesGroupby unique, nsmallest, nlargest (modin-project#63)

* Add client support for SeriesGroupby unique, nsmallest, nlargest

Signed-off-by: Naren Krishna <[email protected]>

---------

Signed-off-by: Naren Krishna <[email protected]>

Push memory_usage entirely to query compiler [change is not to be upstreamed to Modin] (modin-project#66)

* Fix dataframe memory usage.

Signed-off-by: mvashishtha <[email protected]>

* Fix series memory_usage() the same way.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FIX: allow updating backend query compilers in place. (modin-project#67)

* FIX: Mutate client query compiler columns and index in the service.

Motivation: Align axis update semantics across query compilers. In the base
query compiler and even our service's query compiler, you can update the index
and columns in place. However, the service gives no way to update axes of a
query compiler.

Right now, for inplace updates, service exposes an extra method rename(), and
client query compiler uses this to get the id of a new compiler with updated
axis, and then updates its id ID of the new query compiler.

This change might be the first to make the service present a mutable interface
for a backend query compiler. That seems safe to me, except I had to make
copy() get a new query compiler copied from the old query compiler, because we
can't let updates to the new query compiler change the original (or vice versa).

Signed-off-by: mvashishtha <[email protected]>

* Add a comment.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FEAT replace groupby.fillna with a simpler logic (modin-project#68)

* FEAT Support expanding.aggregate

* Replaced groupby.fillna logic with a simpler one

* Fix in groupby.fillna. Work object was causing problems.

* Only need to change _check_index_name to _check_index

* Removed commented out code.
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Feb 27, 2023
@helmeleegy @vnlitvinov
Fix dt_day_of_week/day_of_year (modin-project#51) (service)
2d8a40a
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix str_cat/extract/partition/replace/rpartition (modin-project#51) (…
dc29c53 
…core)
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix dt_day_of_week/day_of_year (modin-project#51) (service)
92f304f
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix str_cat/extract/partition/replace/rpartition (modin-project#51) (…
b081910 
…core)
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix dt_day_of_week/day_of_year (modin-project#51) (service)
d356129
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix dt_day_of_week/day_of_year (modin-project#51) (service)
a567f5a
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix str_cat/extract/partition/replace/rpartition (modin-project#51) (…
c727887 
…core)
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this pull request Mar 16, 2023
@helmeleegy @vnlitvinov
Fix dt_day_of_week/day_of_year (modin-project#51) (service)
a164b91
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simon-mo simon-mo simon-mo approved these changes

Assignees
No one assigned
Labels
bug 🦗 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@devin-petersohn @simon-mo @kunalgosar