dynamic.qmd

---
execute:
  freeze: auto
---

# Dynamic branching {#dynamic}

```{r, message = FALSE, warning = FALSE, echo = FALSE}
knitr::opts_knit$set(root.dir = fs::dir_create(tempfile()))
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = TRUE)
```

```{r, message = FALSE, warning = FALSE, echo = FALSE}
library(targets)
library(tarchetypes)
library(dplyr)
```

:::{.callout-tip}
## Performance

Branched pipelines can be computationally demanding. See the [performance chapter](#performance) for options, settings, and other choices to optimize and monitor large pipelines.
:::

## Branching

Sometimes, a pipeline contains more targets than a user can comfortably type by hand. For projects with hundreds of targets, branching can make the code in `_targets.R` shorter and more concise. 

`targets` supports two types of branching: dynamic branching and [static branching](#static). Some projects are better suited to dynamic branching, while others benefit more from [static branching](#static) or a combination of both. Here is a short list of tradeoffs.

Dynamic | Static
---|---
Pipeline creates new targets at runtime. | All targets defined in advance.
Cryptic target names. | Friendly target names.
Scales to hundreds of branches. | Does not scale as easily for `tar_visnetwork()` etc.
No metaprogramming required. | Familiarity with metaprogramming is helpful.

## About dynamic branching

Dynamic branching is the act of defining new targets (called branches) while the pipeline is running (e.g. during `tar_make()`). Prior to launching the pipeline, the user does not need to know the number of branches or the input data of each branch.

To use dynamic branching, set the `pattern` argument of `tar_target()`. The pattern determines how dynamic branches are created and how the input data is partitioned among the branches. A branch is single iteration of the target's command on a single piece of the input data. Branches are automatically created based on how the input data breaks into pieces, and `targets` automatically combines the output from all the branches when you reference the dynamic target as a whole.

## Example

To illustrate, consider the example pipeline below. It uses dynamic branching to generate random [spirographs](https://en.wikipedia.org/wiki/Spirograph) using code borrowed from [W. Joel Schneider](https://github.com/wjschne)'s [`spiro`](https://wjschne.github.io/spiro/index.html) package.^[The pipeline uses code borrowed from the [`spiro`](https://wjschne.github.io/spiro/index.html) package. The code in the `spirograph_points()` function is adapted from <https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/R/spirograph.R>
under the CC0 1.0 Universal license: <https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/LICENSE.md>]. A spirograph is a type of two-dimensional algebraic curve determined (in part) by parameters `fixed_radius` and `cycling_radius`. Targets `fixed_radius` and `cycling_radius` draw random parameter values, and the dynamic target `points` generates a spirograph dataset for each set of parameters (one spirograph per dynamic branch). Target `single_plot` plots each spirograph separately, and `combine_plot` plots all the spirographs together.

```{r, echo = FALSE, eval = TRUE}
library(targets)
library(tarchetypes)
tar_script({
  library(ggplot2)
  library(targets)
  library(tarchetypes)
  library(tibble)
  
  # From https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/R/spirograph.R
  # Adapted under the CC0 1.0 Universal license: https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/LICENSE.md
  spirograph_points <- function(fixed_radius, cycling_radius) {
    t <- seq(1, 30 * pi, length.out = 1e4)
    diff <- (fixed_radius - cycling_radius)
    ratio <- diff / cycling_radius
    x <- diff * cos(t) + cos(t * ratio)
    y <- diff * sin(t) - sin(t * ratio)
    tibble(x = x, y = y, fixed_radius = fixed_radius, cycling_radius = cycling_radius)
  }
  
  plot_spirographs <- function(points) {
    label <- "fixed_radius = %s, cycling_radius = %s"
    points$parameters <- sprintf(label, points$fixed_radius, points$cycling_radius)
    ggplot(points) +
      geom_point(aes(x = x, y = y, color = parameters), size = 0.1) +
      facet_wrap(~parameters) +
      theme_gray(16) +
      guides(color = "none")
  }
  
  list(
    tar_target(fixed_radius, sample.int(n = 10, size = 2)),
    tar_target(cycling_radius, sample.int(n = 10, size = 2)),
    tar_target(
      points,
      spirograph_points(fixed_radius, cycling_radius),
      pattern = map(fixed_radius, cycling_radius)
    ),
    tar_target(
      single_plot,
      plot_spirographs(points),
      pattern = map(points),
      iteration = "list"
    ),
    tar_target(combined_plot, plot_spirographs(points))
  )
})
```

```{r, eval = FALSE}
# _targets.R
library(ggplot2)
library(targets)
library(tarchetypes)
library(tibble)

# From https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/R/spirograph.R
# Adapted under the CC0 1.0 Universal license: https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/LICENSE.md
spirograph_points <- function(fixed_radius, cycling_radius) {
  t <- seq(1, 30 * pi, length.out = 1e4)
  diff <- (fixed_radius - cycling_radius)
  ratio <- diff / cycling_radius
  x <- diff * cos(t) + cos(t * ratio)
  y <- diff * sin(t) - sin(t * ratio)
  tibble(x = x, y = y, fixed_radius = fixed_radius, cycling_radius = cycling_radius)
}

plot_spirographs <- function(points) {
  label <- "fixed_radius = %s, cycling_radius = %s"
  points$parameters <- sprintf(label, points$fixed_radius, points$cycling_radius)
  ggplot(points) +
    geom_point(aes(x = x, y = y, color = parameters), size = 0.1) +
    facet_wrap(~parameters) +
    theme_gray(16) +
    guides(color = "none")
}

list(
  tar_target(fixed_radius, sample.int(n = 10, size = 2)),
  tar_target(cycling_radius, sample.int(n = 10, size = 2)),
  tar_target(
    points,
    spirograph_points(fixed_radius, cycling_radius),
    pattern = map(fixed_radius, cycling_radius)
  ),
  tar_target(
    single_plot,
    plot_spirographs(points),
    pattern = map(points),
    iteration = "list"
  ),
  tar_target(combined_plot, plot_spirographs(points))
)
```

```{r, eval = TRUE}
tar_make()
```

The final plot shows all the spirographs together.

```{r, eval = TRUE}
tar_read(combined_plot)
```

This plot comes from all the branches of `points` aggregated together. Because target `points` has `iteration = "vector"` in `tar_target()`, any reference to the whole target automatically aggregates the branches using `vctrs::vec_c()`. For data frames, this just binds all the rows.

```{r, eval = TRUE}
tar_read(points)
```

By contrast, target `single_plot` list of branches because of `iteration = "list"`.

```{r}
tar_load(single_plot)
class(single_plot)
length(single_plot)
```

Use the `branches` argument of `tar_read()` to read an individual branch or subset of branches.

```{r}
tar_read(single_plot, branches = 1)
```


## Provenance

Recall our dynamic target `points` with branches for spirograph datasets. Each branch has columns `fixed_radius` and `cycling_radius` so we know which parameter set each spirograph used. It is good practice to proactively append this metadata to each branch, e.g. in `spirograph_points()`. That way, if a branch errors out, it is easy to track down the upstream data that caused it.^[See also <https://books.ropensci.org/targets/debugging.html#workspaces>.]

```{r}
tar_read(points, branches = 1) # first branch
```

```{r}
tar_read(points, branches = 2) # second branch
```

## Patterns

`targets` supports many more types of dynamic branching patterns.

* `map()`: one branch per tuple of elements.
* `cross()`: one branch per *combination* of elements.
* `slice()`: select individual pieces to branch over. For example, `pattern = slice(x, index = c(3, 4))` branches over the third and fourth slices (or branches) of target `x`.
* `head()`: branch over the first few elements.
* `tail()`: branch over the last few elements.
* `sample()`: branch over a random subset of elements.

Patterns are composable. For example, `pattern = cross(other_parameter, map(fixed_radius, cycling_radius))` is conceptually equivalent to `tidyr::crossing(other_parameter, tidyr::nesting(fixed_radius, cycling_radius))`. You can test and experiment with branching structures using [`tar_pattern()`](https://docs.ropensci.org/targets/reference/tar_pattern.html). In the output below, suffixes `_1`, `_2`, and `_3`, denote both dynamic branches and the slices of upstream data they branch over.

```{r, eval = TRUE}
tar_pattern(
  cross(other_parameter, map(fixed_radius, cycling_radius)),
  other_parameter = 3,
  fixed_radius = 2,
  cycling_radius = 2
)
```

## Iteration

The `iteration` argument of `tar_target()` determines how to split non-dynamic targets and how to aggregate dynamic ones. There are two major types of iteration: `"vector"` (default) and `"list"`. There is also `iteration = "group"`, which this chapter covers in the later section on branching over row groups.

### Vector iteration

Vector iteration uses the `vctrs` package to intelligently split and combine dynamic branches based on the underlying type of the object. Branches of vectors are automatically vectors, branches of data frames are automatically data frames, aggregates of vectors are automatically vectors, and aggregates of data frames are automatically data frames. This consistency makes most data processing tasks extremely smooth.

Consider the following pipeline:

```{r, eval = FALSE}
library(targets)
library(tarchetypes)
library(tibble)

list(
  tar_target(
    name = cycling_radius,
    command = c(1, 2),
    iteration = "vector"
  ),
  tar_target(
    name = points_template,
    command = tibble(x = c(1, 2), y = c(1, 2), fixed_radius = c(1, 2)),
    iteration = "vector"
  ),
  tar_target(
    name = points_branches,
    command = add_column(points_template, cycling_radius = cycling_radius),
    pattern = map(cycling_radius, points_template),
    iteration = "vector"
  ),
  tar_target(
    name = combined_points,
    command = points_branches
  )
)
```

```{r, echo = FALSE, eval = TRUE}
library(targets)
library(tarchetypes)
tar_script({
  library(targets)
  library(tarchetypes)
  library(tibble)
  list(
    tar_target(
      name = cycling_radius,
      command = c(3, 4),
      iteration = "vector"
    ),
    tar_target(
      name = points_template,
      command = tibble(x = c(1, 2), y = c(1, 2), fixed_radius = c(1, 2)),
      iteration = "vector"
    ),
    tar_target(
      name = points_branches,
      command = add_column(points_template, cycling_radius = cycling_radius),
      pattern = map(cycling_radius, points_template),
      iteration = "vector"
    ),
    tar_target(
      name = combined_points,
      command = points_branches
    )
  )
})
```

```{r, eval = TRUE}
tar_visnetwork()
```

```{r, eval = TRUE}
tar_make()
```

We observe the following:

```{r, eval = TRUE}
tar_read(points_branches)

tar_read(points_branches, branches = 2)

tar_read(combined_points)
```

`iteration = "vector"` produces convenient `tibble`s because:

1. `vctrs::vec_slice()` intelligently splits the non-dynamic targets for branching.
2. `vctrs::vec_c()` implicitly combines branches when you reference a dynamic target as a whole.

So the pipeline is equivalent to:

```{r, eval = FALSE}
# cycling_radius target:
cycling_radius <- c(3, 4)

# points_template target:
points_template <- tibble(x = c(1, 2), y = c(1, 2), fixed_radius = c(1, 2))

# points_branches target:
points_branches <- lapply(
  X = seq_len(2),
  FUN = function(index) {
    # effect of iteration = "vector" in cycling_radius:
    branch_cycling_radius <- vctrs::vec_slice(cycling_radius, index)
    # effect of iteration = "vector" in points_template:
    branch_points_template <- vctrs::vec_slice(points_template, index)
    # command of points_branches target:
    add_column(branch_points_template, cycling_radius = branch_cycling_radius)
  }
)

# combined_points target:
points_branches$.name_spec = "{outer}_{inner}"
combined_points <- do.call( # effect of iteration = "vector" in points_branches
  what = vctrs::vec_c,
  args = points_branches
)
```

### List iteration

`iteration = "vector"` does not know how to split or aggregate every data type. For example, `vctrs` cannot combine `ggplot2` objects into a vector. `iteration = "list"` is a simple workaround that treats everything as a list during splitting and aggregation. Let's demonstrate on a simple pipeline:

```{r, eval = FALSE}
library(targets)
library(tarchetypes)
library(tibble)

list(
  tar_target(
    name = radius_origin,
    command = c(1, 2),
    iteration = "list"
  ),
  tar_target(
    name = radius_branches,
    command = radius_origin + 5,
    pattern = map(radius_origin),
    iteration = "list"
  ),
  tar_target(
    name = radius_combined,
    command = radius_branches
  )
)
```

```{r, echo = FALSE, eval = TRUE}
library(targets)
library(tarchetypes)
tar_script({
  library(targets)
  library(tarchetypes)
  library(tibble)
  list(
    tar_target(
      name = radius_origin,
      command = c(1, 2),
      iteration = "list"
    ),
    tar_target(
      name = radius_branches,
      command = radius_origin + 5,
      pattern = map(radius_origin),
      iteration = "list"
    ),
    tar_target(
      name = radius_combined,
      command = radius_branches
    )
  )
})
```

```{r, eval = TRUE}
tar_visnetwork()
```

```{r, eval = TRUE}
tar_make()
```

We observe the following:

```{r, eval = TRUE}
tar_read(radius_branches)

tar_read(radius_branches, branches = 2)

tar_read(radius_combined)
```

As we see above, `iteration = "list"` uses `[[` to split non-dynamic targets and `list()` to combine dynamic branches. Except for the special branch names above, our example pipeline is equivalent to:

```{r, eval = TRUE}
# radius_origin target:
radius_origin <- c(1, 2)

# radius_branches target:
radius_branches <- lapply(
  X = seq_len(2),
  FUN = function(index) {
    # effect of iteration = "list" in radius_origin:
    branch_radius_origin <- radius_origin[[index]]
    # command of radius_branches:
    branch_radius_origin + 5
  }
)

# command of radius_combined:
radius_combined <- do.call( # effect of iteration = "list" in radius_branches
  what = list,
  args = radius_branches
)
```

## Branching over row groups

To dynamically branch over `dplyr::group_by()` row groups of a non-dynamic data frame, use `iteration = "group"` together with `tar_group()`. The target with `iteration = "group"` must not already be a dynamic target. (In other words, it is invalid to set `iteration = "group"` and `pattern = map(...)` for the same target.)

To demonstrate group iteration, consider the following alternative version of the spirograph pipeline. Below, we start with a monolithic data frame with all the spirographs together, and then we branch over the row groups of that data frame to create one visual for each dynamic branch.

```{r, echo = FALSE, eval = TRUE}
tar_script({
  # _targets.R
  suppressPackageStartupMessages(library(dplyr))
  library(ggplot2)
  library(targets)
  library(tarchetypes)
  library(tibble)
  
  # From https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/R/spirograph.R
  # Adapted under the CC0 1.0 Universal license: https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/LICENSE.md
  spirograph_points <- function(fixed_radius, cycling_radius) {
    t <- seq(1, 30 * pi, length.out = 1e4)
    diff <- (fixed_radius - cycling_radius)
    ratio <- diff / cycling_radius
    x <- diff * cos(t) + cos(t * ratio)
    y <- diff * sin(t) - sin(t * ratio)
    tibble(x = x, y = y, fixed_radius = fixed_radius, cycling_radius = cycling_radius)
  }
  
  plot_spirographs <- function(points) {
    label <- "fixed_radius = %s, cycling_radius = %s"
    points$parameters <- sprintf(label, points$fixed_radius, points$cycling_radius)
    ggplot(points) +
      geom_point(aes(x = x, y = y, color = parameters), size = 0.1) +
      facet_wrap(~parameters) +
      theme_gray(16) +
      guides(color = "none")
  }
  
  list(
    tar_target(
      points,
      bind_rows(
        spirograph_points(3, 9),
        spirograph_points(7, 2)
      ) %>%
        group_by(fixed_radius, cycling_radius) %>%
        tar_group(),
      iteration = "group"
    ),
    tar_target(
      single_plot,
      plot_spirographs(points),
      pattern = map(points),
      iteration = "list"
    )
  )
})
```

```{r, eval = FALSE}
# _targets.R
library(dplyr)
library(ggplot2)
library(targets)
library(tarchetypes)
library(tibble)

# From https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/R/spirograph.R
# Adapted under the CC0 1.0 Universal license: https://github.com/wjschne/spiro/blob/87f73ec37ceb0a7a9d09856ada8ae28d587a2ebd/LICENSE.md
spirograph_points <- function(fixed_radius, cycling_radius) {
  t <- seq(1, 30 * pi, length.out = 1e4)
  diff <- (fixed_radius - cycling_radius)
  ratio <- diff / cycling_radius
  x <- diff * cos(t) + cos(t * ratio)
  y <- diff * sin(t) - sin(t * ratio)
  tibble(x = x, y = y, fixed_radius = fixed_radius, cycling_radius = cycling_radius)
}

plot_spirographs <- function(points) {
  label <- "fixed_radius = %s, cycling_radius = %s"
  points$parameters <- sprintf(label, points$fixed_radius, points$cycling_radius)
  ggplot(points) +
    geom_point(aes(x = x, y = y, color = parameters), size = 0.1) +
    facet_wrap(~parameters) +
    theme_gray(16) +
    guides(color = "none")
}

list(
  tar_target(
    points,
    bind_rows(
      spirograph_points(3, 9),
      spirograph_points(7, 2)
    ) %>%
      group_by(fixed_radius, cycling_radius) %>%
      tar_group(),
    iteration = "group"
  ),
  tar_target(
    single_plot,
    plot_spirographs(points),
    pattern = map(points),
    iteration = "list"
  )
)
```


```{r}
tar_make()
```

```{r}
tar_read(single_plot, branches = 1)
```

The `tar_group_by()` function in `tarchetypes` makes this branching easier. Using `tar_group_by()`, the pipeline condenses down to:

```{r, eval = FALSE}
list(
  tar_group_by(
    points,
    bind_rows(
      spirograph_points(3, 9),
      spirograph_points(7, 2)
    ),
    fixed_radius,
    cycling_radius
  ),
  tar_target(
    single_plot,
    plot_spirographs(points),
    pattern = map(points),
    iteration = "list"
  )
)
```

For similar functions that branch across row groups, visit <https://docs.ropensci.org/tarchetypes/reference/index.html#dynamic-grouped-data-frames>.


## Branching over files

Dynamic branching over files is tricky. A target with `format = "file"` treats the entire set of files as an irreducible bundle. That means in order to branch over files downstream, each file must already have its own branch. Here is a pipeline that begins with spirograph data files and loads each into a different dynamic branch.

```{r, eval = FALSE}
# _targets.R
library(targets)
library(tarchetypes)
list(
  tar_target(paths, c("spirograph_dataset_1.csv", "spirograph_dataset_1.csv")),
  tar_target(files, paths, format = "file", pattern = map(paths)),
  tar_target(data, read_csv(files), pattern = map(files))
)
```

The [`tar_files()`](https://docs.ropensci.org/tarchetypes/reference/tar_files.html) function from the [`tarchetypes`](https://github.com/ropensci/tarchetypes) package is shorthand for the first two targets above.

```{r, eval = FALSE}
# _targets.R
library(targets)
library(tarchetypes)
list(
  tar_files(files, c("spirograph_dataset_1.csv", "spirograph_dataset_1.csv")),
  tar_target(data, read_csv(files), pattern = map(files))
)
```

## Performance and batching

Dynamic branching makes it easy to create many targets. Unfortunately, if the number of targets exceeds several thousand, overhead may build up and the package may slow down. Temporary workarounds can avoid overhead in specific cases: for example, the `shortcut` argument of `tar_make()`, and choosing a pattern like `slice()` or `head()` instead of a full `map()`. But to minimize overhead at scale, it is better to accomplish the same amount of work with a fewer number of targets. In other words, do more work inside each dynamic branch.

Batching is particularly useful to reduce overhead. In batching, each dynamic branch performs multiple computations instead of just one. The `tarchetypes` package supports several general-purpose functions that do batching automatically: most notably `tar_rep()` and `tar_map_rep()` for simulation studies and `tar_group_count()`, `tar_group_size()`, and `tar_group_select()` for batching over the rows of a data frame.

The packages in the [R Targetopia](https://wlandau.github.io/targetopia/) support batching for specific use cases. For example, in [`stantargets`](https://docs.ropensci.org/stantargets/index.html), [`tar_stan_mcmc_rep_summary()`](https://docs.ropensci.org/stantargets/reference/tar_stan_mcmc_rep_summary.html) dynamically branches over batches of simulated datasets for Stan models.

The [`targets-stan`](https://github.com/wlandau/targets-stan) repository has an example of custom batching implemented from scratch. The goal of the pipeline is to validate a Bayesian model by simulating thousands of dataset, analyzing each with a Bayesian model, and assessing the overall accuracy of the inference. Rather than define a target for each dataset in model, the pipeline breaks up the work into batches, where each batch has multiple datasets or multiple analyses. Here is a version of the pipeline with 40 batches and 25 simulation reps per batch (1000 reps total in a pipeline of 82 targets).

```{r, eval = FALSE}
# _targets.R
library(targets)
library(tarchetypes)
list(
  tar_target(model_file, compile_model("stan/model.stan"), format = "file"),
  tar_target(index_batch, seq_len(40)),
  tar_target(index_sim, seq_len(25)),
  tar_target(
    data_continuous,
    purrr::map_dfr(index_sim, ~simulate_data_continuous()),
    pattern = map(index_batch)
  ),
  tar_target(
    fit_continuous,
    map_sims(data_continuous, model_file = model_file),
    pattern = map(data_continuous)
  )
)
```