collect throws an error in a hub without any model output files

[elray1]

If I run the following in a hub that contains no model output files:

library(hubUtils)
library(dplyr)

model_outputs <- hubUtils::connect_hub(hub_path = ".") %>%
  dplyr::collect()

I get this error:

Error in `UseMethod()`:
! no applicable method for 'collect' applied to an object of class "c('hub_connection', 'list')"
Backtrace:
 1. hubUtils::connect_hub(hub_path = ".") %>% dplyr::collect()
 2. dplyr::collect(.)

[annakrystalli]

Thanks for the report @elray1 ! This is a result of a conscious prior decision to not error when connecting to an empty hub. Using connect_hub on an empty hub still loads config metadata but the subclass is an empty list instead of an arrow dataset. Hence dplyr::collect doesn't have a method for it and fails.

This would likely be non-ideal to fix from within connect_hub() but it made me think of a somewhat related discussion @nickreich had raised a while ago (transferred to an issue in this repo: #13 ) in which the ability to run as_model_out_tbl() on the results of collect by default was proposed through a collect_hub() function. Such a function would be an ideal place to handle the empty hub exception too.

Note that one thing we could also do is rather than have a different function (collect_hub()) we could have a collect() S3 method of our own that is dispatched on hub-connection class objects.

[elray1]

I like the idea of our own collect method

[annakrystalli]

In the end, trying to handle this in a hubverse collect() method won't work sadly because collect() may well be preceded by queries, e.g. filter()

hub_path <- system.file("testhubs/empty", package = "hubUtils")
hub_con <- hubData::connect_hub(hub_path) |> 
    dplyr::filter(is.na(output_type_id)) |>
    dplyr::collect()
#> Warning in hubData::connect_hub(hub_path): No files of file formats "csv", "parquet", and "arrow" found in model output
#> directory.
#> Error in UseMethod("filter"): no applicable method for 'filter' applied to an object of class "c('hub_connection', 'list')"

^{Created on 2024-03-27 with reprex v2.0.2}

But in both filtering and just collecting, I'm getting an informative warning from connect_hub() that the hub is effectively empty. Is that not helpful enough? Or are you for some reason not getting that warning @elray1 ?

hub_path <- system.file("testhubs/empty", package = "hubUtils")
hub_con <- hubData::connect_hub(hub_path) |> dplyr::collect()
#> Warning in hubData::connect_hub(hub_path): No files of file formats "csv", "parquet", and "arrow" found in model output
#> directory.
#> Error in UseMethod("collect"): no applicable method for 'collect' applied to an object of class "c('hub_connection', 'list')"

^{Created on 2024-03-27 with reprex v2.0.2}