A framework for parallelizing dependent tasks
Author: Jeremy Coyle
delayed is an R package that provides a framework for parallelizing
dependent tasks in an efficient manner. It brings to R a subset of the
functionality implemented in Python’s Dask
library. For details on how best to
use delayed, please consult the package
documentation and
vignette
online, or do so from within R.
For standard use, we recommend installing the package from CRAN via
install.packages("delayed")Install the most recent stable release from GitHub via
devtools:
devtools::install_github("tlverse/delayed")If you encounter any bugs or have any specific feature requests, please file an issue.
This minimal example shows how to use delayed to handle dependent
computations via chaining of tasks:
library(delayed)
#> delayed v0.4.0: A Framework for Parallelizing Dependent Tasks
# delay a function that does a bit of math
mapfun <- function(x, y) {(x + y) / (x - y)}
delayed_mapfun <- delayed_fun(mapfun)
set.seed(14765)
library(future)
plan(multicore, workers = 2)
const <- 7
# re-define the delayed object from above
delayed_norm <- delayed(rnorm(n = const))
delayed_pois <- delayed(rpois(n = const, lambda = const))
chained_norm_pois <- delayed_mapfun(delayed_norm, delayed_pois)
# compute it using the future plan (multicore with 2 cores)
chained_norm_pois$compute(nworkers = 2, verbose = TRUE)
#> run:0 ready:2 workers:2
#> updating rnorm(n = const) from ready to running
#> run:1 ready:1 workers:2
#> updating rpois(n = const, lambda = const) from ready to running
#> run:2 ready:0 workers:2
#> updating rnorm(n = const) from running to resolved
#> updating rpois(n = const, lambda = const) from running to resolved
#> updating mapfun(x = delayed_norm, y = delayed_pois) from waiting to ready
#> run:0 ready:1 workers:2
#> updating mapfun(x = delayed_norm, y = delayed_pois) from ready to running
#> run:1 ready:0 workers:2
#> updating mapfun(x = delayed_norm, y = delayed_pois) from running to resolved
#> [1] -0.6688907 -1.2691496 -1.1808899 -1.7605806 -0.5992127 -0.6838026 -1.4086257Remark: In the above, the delayed computation is carried out in
parallel using the framework offered by the excellent future
package and its associated
ecosystem.
© 2017-2021 Jeremy R. Coyle
The contents of this repository are distributed under the GPL-3 license.
See file LICENSE for details.