Which branching function(s) is best for multiple model workflows?

[JakeRuss]

Hi Will,

I'm trying to convert a long tidymodels script into a targets pipeline; your targets-keras example has been a great help in this regard. The key difference between that example and my use case is that I want to fit a bunch of identical model workflows to different outcome variables. Take all the same features used to predict churn and run all of those same modeling steps, but on both churn and lifetime value (LTV, to stick with the saas example).

I've set up my targets pipeline as follows,

list(
  tar_target(survey_model_data, query_model_columns(project_name),       format = "aws_rds"),
  tar_target(model_outcome_names, distinct(survey_model_data, model_name) %>% pull(model_name), format = "aws_rds"),
  tar_target(training_set_list, create_modeling_data(survey_model_data), format = "aws_rds"),
  tar_map() ?
)

training_set_list is a named-list of identical data frames (id, outcome, x1, x2, ...xN) only the outcome column is different and I want to send each data frame into its own workflow: split, recipe, grid search, fit best, ...etc. I could write wrapper functions that map, for example, initial_split over the list of data frames and return yet another list containing split objects, but I believe that approach would not harness one of the best features of targets, the ability to parallelize tasks. Instead of creating one large aggregate list-target per step, I'd like to use branching to run inital_split on each data frame as separate tasks and then pass those splits to recipes and so on.

Do you mind providing advice here?

[wlandau]

Since training_set_list is a target, dynamic branching seems like a reasonable thing to try: https://books.ropensci.org/targets/dynamic.html. (And because of the structure of training_set_list, you might also consider iteration = "list" for that target.) Since you are familiar with the Keras example, you might check out the interactive exercises in the branching chapter of the short course.

[wlandau]

Also, "aws_qs" might be more efficient for large data than "aws_rds".

Sketch:

# _targets.R
library(targets)
run_model <- function(training_set_list) {
  # ...
}
list(
  tar_target(survey_model_data, query_model_columns(project_name),       format = "aws_qs"),
  tar_target(model_outcome_names, distinct(survey_model_data, model_name) %>% pull(model_name), format = "aws_qs"),
  tar_target(training_set_list, create_modeling_data(survey_model_data), format = "aws_qs", iteration = "list"),
  tar_target(model, run_model(training_set_list), pattern = map(training_set_list), format = "aws_qs")
)

[JakeRuss]

I wish I had found that short course chapter before asking, thanks Will, your feedback has me unstuck now. The iteration = "list" is exactly what I was looking for