Visual Test Framework for Static Charts

[jwildfire]

Thanks again for #34 , @rescalan . It was ready to merge, and I have a few thoughts to follow. I feel no one looks at closed PRs, so I thought it might be good to follow up and chat in a thread here.

I've never used {vdiffr}, but we played with snapshot based tests in {shinytest} in v1, but eventually abandoned it for 2 main reasons:

1. We constantly had minor differences in snapshots on different operating systems. In particular, we could never get tests to pass in Travis CI.
2. The image files were causing problems with the 5mb file size limitation when submitting the package to CRAN.

To be fair, we didn't really try that hard to get things working, but just wanted to see if you or others have experience with either issue.

[rescalan]

The authors of {vdiffr} seem to have added a more robust figure generation system across platforms in version 0.3.0.

From the news

vdiffr now works reliably across platforms:

svglite is now embedded in vdiffr to protect against updates of the SVG generation engine.

However they do recognize that this framework is fragile

Despite our efforts to make vdiffr robust and reliable across platforms, checking the appearance of a figure is still inherently fragile. It is similar to testing for errors by matching exact error messages: these messages are susceptible to change at any time. Similarly, the appearance of plots depends on a lot of upstream code, such as the way margins and spacing are computed. vdiffr uses a special ggplot2 theme that should change very rarely, but there are just too many upstream factors that could cause breakages. For this reason, figure mismatches are not necessarily representative of actual failures.

@jwildfire Is the image size limitation for CRAN of 5 MB per file or all files in the package?

[jwildfire]

5mb for the package 🤷‍♂️ . Though seems like maybe we can dodge that issue by not running those checks on CRAN per your comment in chat.