Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fillna with dictionary assumes each column/row gets its own partition #50

Closed
devin-petersohn opened this issue Jul 20, 2018 · 1 comment
Closed

fillna with dictionary assumes each column/row gets its own partition #50

devin-petersohn opened this issue Jul 20, 2018 · 1 comment
Assignees
@devin-petersohn
Labels
bug 🦗 Something isn't working

Comments

@devin-petersohn
Copy link
Collaborator

Describe the problem

This was discovered when I was working on #49.

Source code / logs

Setting npartitions to something less than 4 will break the test_fillna_sanity test when it tries to apply the dictionary replacement.
@devin-petersohn devin-petersohn added the bug 🦗 Something isn't working label Jul 20, 2018
@devin-petersohn devin-petersohn self-assigned this Jul 20, 2018
@devin-petersohn devin-petersohn mentioned this issue Jul 20, 2018
Merged
1 task
@simon-mo
Copy link
Collaborator

Closed by #51

dchigarev pushed a commit to dchigarev/modin that referenced this issue Aug 25, 2020
@ienkovich
Merge pull request modin-project#50 from intel-go/ienkovich/cat-codes…
0084e8d 
…-null

fix cat.codes for NULLs
mvashishtha pushed a commit to mvashishtha/modin that referenced this issue Jan 30, 2023
@batur-ponder @helmeleegy
POND-815 fixes for 21 column dataset (modin-project#50)
2992548 
* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this issue Feb 13, 2023
@batur-ponder @helmeleegy @vnlitvinov
POND-815 fixes for 21 column dataset (modin-project#50)
3e88b98 
* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>
RehanSD pushed a commit to RehanSD/modin that referenced this issue Feb 17, 2023
@devin-petersohn
FEAT-modin-project#4931: Create a query compiler that can connect to …
20653a4 
…a service

Signed-off-by: Devin Petersohn <[email protected]>

Fixes to pass CI + docs for io.py

Update implementation

Signed-off-by: Devin Petersohn <[email protected]>

Fix some things

Signed-off-by: Devin Petersohn <[email protected]>

Lint fixes

Fix put

Signed-off-by: Devin Petersohn <[email protected]>

Clean up and add new details

Signed-off-by: Devin Petersohn <[email protected]>

Use fsspec to get full path and allow URLs

Signed-off-by: Devin Petersohn <[email protected]>

Add lazy loc

Signed-off-by: Devin Petersohn <[email protected]>

fixes for tests

porting more tests

more fixes

moar fixes

Raise exception

Signed-off-by: Devin Petersohn <[email protected]>

Lint fixes

Return Python as the default modin engine

Handle indexing case for client qc

Call fast path for __getitem__ if not lazy

Remove user warning for Python-engine fall back

Add init

Signed-off-by: Devin Petersohn <[email protected]>

Implement free as a no-op

Signed-off-by: Devin Petersohn <[email protected]>

Add support for replace - client side

Fix a couple of issues with Client

Signed-off-by: Devin Petersohn <[email protected]>

Throw errors on to_pandas

Signed-off-by: Devin Petersohn <[email protected]>

Do not default to pandas for str_repeat

Add support for 18 datetime functions/properties

Fix columns caching when renaming columns

Fix test_query: put backticks back for col names

Add support for astype -- client side

hard coded changes for functions

Client support for str_(en/de)code, to_datetime

Add all missing query compiler methods.

Signed-off-by: mvashishtha <[email protected]>

Fix getitem_column_array and take_2d.

Signed-off-by: mvashishtha <[email protected]>

Fix getitem_column_array and take_2d.

Signed-off-by: mvashishtha <[email protected]>

Fix again.

Signed-off-by: mvashishtha <[email protected]>

Fix more bugs.

Signed-off-by: mvashishtha <[email protected]>

More fixes.

Signed-off-by: mvashishtha <[email protected]>

Fix more bugs-- pushdown tests test_dates and test_pivot still broken due to service bugs.

Signed-off-by: mvashishtha <[email protected]>

Fix typo. Note drop() broken because service requires you to specify both argument and client QC at base of this PR uses default Nones.

Signed-off-by: mvashishtha <[email protected]>

Add query compiler class.

Signed-off-by: mvashishtha <[email protected]>

Testing a commit

Initial changes for adding support for Expanding

FEAT Support for rolling.sem

FEAT support for Expanding sum, min, max, mean, var, std, count, sem

Removing extratenous comment

REFACTOR: Remove defaults to pandas at API layer and add some corresponding client QC methods.

Signed-off-by: mvashishtha <[email protected]>

Add more methods.

Signed-off-by: mvashishtha <[email protected]>

Fix expanding.

Signed-off-by: mvashishtha <[email protected]>

Add ewm.

Signed-off-by: mvashishtha <[email protected]>

Revert whitespace.

Signed-off-by: mvashishtha <[email protected]>

Fix to_numpy by making it like to_pandas.

Signed-off-by: mvashishtha <[email protected]>

Remove extra to_numpy.

Signed-off-by: mvashishtha <[email protected]>

Pass kwargs

Signed-off-by: mvashishtha <[email protected]>

Fix DataFrame import for isin.

Signed-off-by: mvashishtha <[email protected]>

Fix again.

Signed-off-by: mvashishtha <[email protected]>

Remove breakpoint

Signed-off-by: mvashishtha <[email protected]>

Tell if series.

Signed-off-by: mvashishtha <[email protected]>

Fix client qc.

Signed-off-by: mvashishtha <[email protected]>

Add self_is_series.

Signed-off-by: mvashishtha <[email protected]>

FIX: Set numeric_only to True in groupby quantile

Add some comments

Fix str_cat/fullmatch/removeprefix/removesuffix/translate/wrap (modin-project#44)

* Fix str_cat/fullmatch/removeprefix/removesuffix/translate/wrap

* Update modin/core/storage_formats/base/query_compiler.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

* Update modin/pandas/series_utils.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

* Update modin/core/storage_formats/base/query_compiler.py

Co-authored-by: Mahesh Vashishtha <[email protected]>

Co-authored-by: Mahesh Vashishtha <[email protected]>

FEAT Support expanding.aggregate (modin-project#45)

Fix at_time and between_time. (modin-project#43)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

Add QC method for groupby.sem (modin-project#47)

* FEAT: Add partial support for groupby.sem()

* Add sem changes to groupby

Fix nlargest and nsmallest Series support (modin-project#46)

* Fix nlargest and smallest support

Signed-off-by: Naren Krishna <[email protected]>

Remove client query compiler's columnarize. (modin-project#48)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

Fix info and set memory_usage=False. (modin-project#49)

Signed-off-by: mvashishtha <[email protected]>

Signed-off-by: mvashishtha <[email protected]>

POND-815 fixes for 21 column dataset (modin-project#50)

* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>

Bring in upstream series binary operation fix 6d5545f… (modin-project#52)

* Bring in upstream series binary operation fix 6d5545f.

Signed-off-by: mvashishtha <[email protected]>

* Update modin/pandas/series.py

Co-authored-by: Karthik Velayutham <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>
Co-authored-by: Karthik Velayutham <[email protected]>

Support groupby first/last (modin-project#53)

Signed-off-by: Naren Krishna <[email protected]>

FEAT: Add initial partial support for groupby.cumcount() (modin-project#54)

* FEAT: Add partial support for cumcount

* Remove the set_index_name

* Squeeze the result

* Write cumcount name to None

* Can't set dtype to int64

Fix resample sum, prod, size (modin-project#56)

Signed-off-by: Naren Krishna <[email protected]>

POND-184: fix describe and simplify query compiler interface (modin-project#55)

* Fix describe

Signed-off-by: mvashishtha <[email protected]>

* Pass datetime_is_numeric.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)

* Fix dt_day_of_week/day_of_year, str_partition/replace/rpartition

* Fix str_extract

Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)" (modin-project#58)

This reverts commit f7a31ab.

Revert "Revert "Fix dt_day_of_week/day_of_year, str_cat/extract/partition/replace/rpartition (modin-project#51)" (modin-project#58)" (modin-project#60)

This reverts commit ad9231d.

Add query compiler method for groupby.prod() (modin-project#57)

Signed-off-by: Naren Krishna <[email protected]>

FEAT: Add support for groupby.head and groupby.tail (modin-project#61)

* FEAT: Add support for groupby.head and groupby.tail

* Change _change_index

FEAT: Add partial support for groupby.nth (modin-project#62)

FIX: Push first and last down to query compiler. (modin-project#64)

* FIX: Push first and last down to query compiler.

Signed-off-by: mvashishtha <[email protected]>

* Fix last.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FEAT: Add partial support for groupby.ngroup (modin-project#65)

* FEAT: Add partial support for groupby.ngroup

* Name of result should be none for now

Add client support for SeriesGroupby unique, nsmallest, nlargest (modin-project#63)

* Add client support for SeriesGroupby unique, nsmallest, nlargest

Signed-off-by: Naren Krishna <[email protected]>

---------

Signed-off-by: Naren Krishna <[email protected]>

Push memory_usage entirely to query compiler [change is not to be upstreamed to Modin] (modin-project#66)

* Fix dataframe memory usage.

Signed-off-by: mvashishtha <[email protected]>

* Fix series memory_usage() the same way.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FIX: allow updating backend query compilers in place. (modin-project#67)

* FIX: Mutate client query compiler columns and index in the service.

Motivation: Align axis update semantics across query compilers. In the base
query compiler and even our service's query compiler, you can update the index
and columns in place. However, the service gives no way to update axes of a
query compiler.

Right now, for inplace updates, service exposes an extra method rename(), and
client query compiler uses this to get the id of a new compiler with updated
axis, and then updates its id ID of the new query compiler.

This change might be the first to make the service present a mutable interface
for a backend query compiler. That seems safe to me, except I had to make
copy() get a new query compiler copied from the old query compiler, because we
can't let updates to the new query compiler change the original (or vice versa).

Signed-off-by: mvashishtha <[email protected]>

* Add a comment.

Signed-off-by: mvashishtha <[email protected]>

---------

Signed-off-by: mvashishtha <[email protected]>

FEAT replace groupby.fillna with a simpler logic (modin-project#68)

* FEAT Support expanding.aggregate

* Replaced groupby.fillna logic with a simpler one

* Fix in groupby.fillna. Work object was causing problems.

* Only need to change _check_index_name to _check_index

* Removed commented out code.
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this issue Mar 16, 2023
@batur-ponder @helmeleegy @vnlitvinov
POND-815 fixes for 21 column dataset (modin-project#50)
66c7dfe 
* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this issue Mar 16, 2023
@batur-ponder @helmeleegy @vnlitvinov
POND-815 fixes for 21 column dataset (modin-project#50)
e3be558 
* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>
vnlitvinov pushed a commit to vnlitvinov/modin that referenced this issue Mar 16, 2023
@batur-ponder @helmeleegy @vnlitvinov
POND-815 fixes for 21 column dataset (modin-project#50)
d98dec1 
* POND-815 fixes for 21 column dataset

* Update modin/pandas/base.py

Co-authored-by: helmeleegy <[email protected]>

---------

Co-authored-by: helmeleegy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@devin-petersohn devin-petersohn

Labels
bug 🦗 Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@devin-petersohn @simon-mo