Add collect_hub function

[annakrystalli]

This PR adds a collect_hub() function which wraps dplyr::collect() but also converts the output to a model_out_tbl class object by default where possible. The function also accepts additional arguments that can be passed to as_model_out_tbl().

Th PR resolves:

• Create hub_connection collect method #17
• On calling connect_hub, default arrow install yields Error: This build of the arrow package does not support Datasets #15
• collect throws an error in a hub without any model output files #4
• data loading and validation #13

I've also modified the R CMD Check workflow to run nightly so that we can pick up any issues arising from upgrades in dependencies promptly. Let's test it out and I can roll it out to all our mature packages when ready.

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 87.23%. Comparing base (18ff938) to head (ca2eec0).

Files	Patch %	Lines
R/collect_hub.R	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #18      +/-   ##
==========================================
+ Coverage   86.98%   87.23%   +0.25%     
==========================================
  Files           9       10       +1     
  Lines         676      705      +29     
==========================================
+ Hits          588      615      +27     
- Misses         88       90       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

🚀 Deployed on https://660acaa966b233bc1dc1cffd--hubdata-pr-preview.netlify.app

[annakrystalli]

The macos latest build is failing because of the temporary issue described in #15 . Not sure what to do about it. I could fix it by installing from the Apache R Universe version in the workflow. That's not where most users would do though so, until we swapped it back to CRAN, we
a. wouldn't get errors representative of median user experience
b. I won't know when the issue is fixed unless I explictly track when a macOS arrow fix is pushed to CRAN.

Happy to hear people's thoughts.

[bsweger]

Leaving this here for other reviewers who might want to fetch this feature branch and give it a spin locally. Where hubverse-infrastructure-test is an S3 bucket with valid hub-config contents and an empty model-output directory:

dplyr::collect() 😢

> load_all()
ℹ Loading hubData
> hub_bad_path <- s3_bucket('hubverse-infrastructure-test')
> hubData::connect_hub(hub_path = hub_bad_path) %>% dplyr::collect()
Error in UseMethod("collect") : 
  no applicable method for 'collect' applied to an object of class "c('hub_connection', 'list')"
In addition: Warning message:
In hubData::connect_hub(hub_path = hub_bad_path) :
  No files of file formats "csv", "parquet", and "arrow" found in model output directory.

hub_collect() 😀

> tbl <- connect_hub(hub_bad_path) %>% collect_hub()
Warning messages:
1: In connect_hub(hub_bad_path) :
  No files of file formats "csv", "parquet", and "arrow" found in model output directory.
2: Hub is empty. No data to collect. Returning `NULL` 

[bsweger]

Thanks, @annakrystalli. Glad we're trying to handle the hard bits on behalf of users!

One or two inline notes, but nothing that would prevent rolling out this improvement.

[bsweger]

Makes sense! Would love to get to a place where these kinds of small operational changes can be in separate PRs so we can get 'em merged in without waiting for review of new features.

[bsweger]

Now that we have collect_hub(), is there any reason someone would want to use dplyr collect()?

As someone with less R proficiency than many folks on the team, I'm left wondering what to do when presented with multiple options like this. Is it worth recommending a default?

[annakrystalli]

collect_hub is mainly a wrapper around dplyr::collect() (a very well known tidyverse function) with some extras. It depends on what they are doing with their data next but there is no reason they must use collect_hub(), it just conveniently outputs a model_out_tbl which many downstream hubverse package functions expect.

I've refactored the article a bit to bring more attention to the benefits of collect_hub and also used it in the connect_hub examples but ultimately collect will work just as well. It just might need an extra step to coerce data to model_out_tbl if used in downstream hubverse functionality