[FEA] add tests on image characteristics

[jameslamb]

Is your feature request related to a problem? Please describe.

Images published from this repo are intended for use in a wide variety of contexts. For example, they're used in several different cloud providers' bring-your-own container solutions (see https://docs.rapids.ai/deployment/stable/).

Those environments each have slightly different requirements for images you bring to them... things like default user, exposed ports, expected filepaths, etc.

When changes like #449 are made to modify the images here, there are currently not strong automatic protections against accidental regressions.

This issue proposes adding such protections, to improve release confidence and hopefully catch some types of issues during development instead of further downstream.

Describe the solution you'd like

I'm proposing the following:

• on every commit to PRs or main here, test all built images with container-canary (https://github.com/NVIDIA/container-canary), with the following tests:
  • compliance with various target deployment environments (with tests sourced from https://github.com/NVIDIA/container-canary/tree/main/examples)
  • other RAPIDS-specific tests to prevent accidental regressions (with tests checked into source control here)
• on every commit to PRs, test files in this repo with some static analyzers. For example:
  • hadolint for Dockerfiles (link)
  • shellcheck for shell scripts (link)
  • ruff for Python code (link)

Describe alternatives you've considered

N/A

Additional context

This proposal comes out of mine and @jacobtomlinson 's discussions around this: rapidsai/deployment#277

And would help with stuff like this:

• [FEA] Allow writing to /var/run #582

I'd be happy to come do the work, just looking for a 👍🏻 / 👎🏻 on the general idea (cc @raydouglass @ajschmidt8 @AyodeAwe )

[jameslamb]

Based on my experiments in #670 so far, I think this will require some updates to container-canary:

• feature request: make container startup timeout configurable NVIDIA/container-canary#61
• feature request: allow overriding command without modifying checks config file NVIDIA/container-canary#62

I'd like to try contributing those, at some point.