-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch parts of overline to data.table #517
Conversation
Test done with
So 257284 segments in Edinburgh |
Sounds promising. If I recall correctly I was seeing more than a 2x speed-up in tests I was doing a few weeks ago: https://github.com/Robinlovelace/overline-tests Just opened-up, was aiming for the tests to be a bit more ready before. 👍 to speeding-up overline! |
@Robinlovelace I've now checked this and got the multicore support working, although it seems to matter a lot less with |
Perhaps I'm not understanding, but I only see small speedups on some of the faster parts of the process? |
Will take a look. |
You can use this branch with remotes::install_github("ropensci/stplanr@overline-dt") |
5x reduction in memory use = v. promising. |
@Robinlovelace @mem48 FYI After a heap of CRAN rejections for one of my pkgs with lots of data.table usage, I learnt that new CRAN checks mean that I needed this line at the top of all examples, tests, and vignettes: data.table::setDTthreads(1L) Without that, CRAN autochecks always rejected because of "ratio of CPU to elaped time > X". In case that helps. |
Handy. Probably over-due a CRAN release. Thanks Mark! |
@Robinlovelace a very quick look at speeding up overline2. I've replaced some
dplyr
code withdata.table
Resutls come out in a different order, but I don't think that matters. I've not tested all use cases like large data and use multicore.
Its a modest speed inmprovements but a major reduction in memory usage which should help.
There is another bit of
dplyr
code I can't work out how to do indata.table
but the fix I have made is to only make one object
sls
rather than making anslg
object that is never used again.