Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DTable] Loading .csv through the loader_function constructor takes allocates double the memory it's supposed to #320

Closed
krynju opened this issue Dec 16, 2021 · 2 comments
Closed

[DTable] Loading .csv through the loader_function constructor takes allocates double the memory it's supposed to #320

krynju opened this issue Dec 16, 2021 · 2 comments
Assignees
@krynju
Labels
performance table interface

Comments

@krynju
Copy link
Member

krynju commented Dec 16, 2021 

julia> d = DTable(CSV.File, readdir("data", join=true)) # 8GB of data
DTable with 500 partitions
Tabletype: unknown (use `tabletype!(::DTable)`)

image
@krynju krynju self-assigned this Dec 16, 2021
@krynju
Copy link
Member Author

krynju commented Dec 18, 2021 •
edited
Loading

This is a CSV.jl issue, same happens with DataFrames.jl
Leaving open for tracking as this sticky memory is not nice for the DTable

@krynju krynju removed the bug label Dec 18, 2021
@krynju
Copy link
Member Author

krynju commented Dec 18, 2021

Ok, not even a CSV.jl issue, but a glibc issue

tracking here: JuliaData/CSV.jl#952

closing this as it has nothing to do with Dagger

@krynju krynju closed this as completed Dec 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krynju krynju

Labels
performance table interface
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@krynju