test: create a test suite for examples

[yoavz]

Description

This PR creates a test suite associated with examples. It runs a minimal test on each example using --local --test mode. It is slightly more heavyweight than our unit tests since every example runs in a separate Python process, but more portable than full integration tests since no Determined cluster is required.

Run-time on my local macbook pro is 1 minute 42 seconds:

41.38s call     test_examples.py::test_official[official/cifar10_cnn_tf_keras-official/cifar10_cnn_tf_keras/const.yaml]
22.36s call     test_examples.py::test_official[official/cifar10_cnn_pytorch-official/cifar10_cnn_pytorch/const.yaml]
9.46s call     test_examples.py::test_official[official/multiple_lr_schedulers_pytorch-official/multiple_lr_schedulers_pytorch/const.yaml]
9.39s call     test_examples.py::test_official[official/mnist_pytorch-official/mnist_pytorch/const.yaml]
8.35s call     test_examples.py::test_official[official/mnist_estimator-official/mnist_estimator/const.yaml]
6.80s call     test_examples.py::test_official[official/fashion_mnist_tf_keras-official/fashion_mnist_tf_keras/const.yaml]
4.51s call     test_examples.py::test_official[official/iris_tf_keras-official/iris_tf_keras/const.yaml]

The runtime can likely further be improved by further optimizing --local --test mode with https://determinedai.atlassian.net/browse/DET-2931

DET-3110
DET-2952

Test Plan

N/A

Commentary (optional)

I had initially aimed to set-up these as unit tests running sequentially in a single Python process with test_one_batch. However, I hit some problems with conflicting module namespaces in examples -- for example, fashion_mnist_tf_keras and mnist_pytorch have a data.py module they import in their entrypoint module with import data. When executed sequentially in a single Python process, the second time import data is called it still uses the initial data module and fails. I figured it would be easier to execute each model definition in a separate process instead of removing this restriction or hacking the Python module namespace in some way.

[brainhart]

@yoavz let's add this to CircleCI?

[yoavz]

@brain-good this is an initial pass at having "unit" tests for examples, with the caveat that I wasn't quite able to get them running all in the same Python process.

Let me know if this approach looks good to you and I'll add the Circle CI target and remove the redundant tests in e2e_tests.

[brainhart]

Sounds good @yoavz , think the approach makes sense 👍

[brainhart]

good stuff