Integration tests for Katib experiments

[i-chvets]

Description

Integration tests for Katib experiments need to be created to ensure all Katib images are functioning as expected.
Integration tests should be a part of bundle test which already has a test for creating a single experiment.
These integration tests should cover the following Katib container images:

• katib/cmd/earlystopping/medianstop/v1beta1
• katib/cmd/metricscollector/v1beta1/tfevent-metricscollector
• katib/cmd/metricscollector/v1beta1/file-metricscollector
• katib/cmd/suggestion/hyperopt/v1beta1
• katib/cmd/suggestion/hyperband/v1beta1
• katib/cmd/suggestion/pbt/v1beta1
• katib/cmd/suggestion/goptuna/v1beta1
• katib/cmd/suggestion/skopt/v1beta1
• katib/cmd/suggestion/optuna/v1beta1
• katib/cmd/suggestion/nas/darts/v1beta1
• katib/cmd/suggestion/nas/enas/v1beta1

Design

TO-DO: Add any related design points here.

Testing

[kimwnasptd]

@i-chvets after taking an extensive look on the suggestion images of Katib I see the following pattern:

• All of the images are just a server
• The server doesn't serve any static files, it's just an executable
• They don't require any K8s context
• Don't require to communicate with any other service (i.e. KFP server needs to talk to db)

So I would suggest that it would be OK for these ROCKs to just have regular smoke tests, that only verify that the servers can start. WDYT?

[i-chvets]

Sound goods good to me, as long as we have a way to verify that server starts for each image, i.e. hit metrics or some other endpoint.
What about these?
katib/cmd/earlystopping/medianstop/v1beta1
katib/cmd/metricscollector/v1beta1/tfevent-metricscollector
katib/cmd/metricscollector/v1beta1/file-metricscollector

[kimwnasptd]

For katib/cmd/earlystopping/medianstop/v1beta1 we can reuse a yaml they already have in https://github.com/kubeflow/katib/blob/master/examples/v1beta1/early-stopping/median-stop.yaml#L19

And for katib/cmd/metricscollector/v1beta1/file-metricscollector we can use the yaml they have in https://github.com/kubeflow/katib/blob/master/examples/v1beta1/metrics-collector/file-metrics-collector.yaml

For the last one katib/cmd/metricscollector/v1beta1/tfevent-metricscollector upstream is testing with Training Operator. Since this could complicate a lot our CI we can postpone this one for now

[NohaIhab]

Based on katib-config ConfigMap, the following examples utilize these images:

Image	upstream example
katib/cmd/earlystopping/medianstop/v1beta1	median-stop.yaml
katib/cmd/metricscollector/v1beta1/tfevent-metricscollector	tfjob-mnist-with-summaries.yaml
katib/cmd/metricscollector/v1beta1/file-metricscollector	file-metrics-collector.yaml
katib/cmd/suggestion/hyperopt/v1beta1	random.yaml
katib/cmd/suggestion/hyperband/v1beta1	hyperband.yaml
katib/cmd/suggestion/pbt/v1beta1	simple-pbt.yaml
katib/cmd/suggestion/goptuna/v1beta1	cma-es.yaml
katib/cmd/suggestion/skopt/v1beta1	bayesian-optimization.yaml
katib/cmd/suggestion/optuna/v1beta1	grid.yaml
katib/cmd/suggestion/nas/darts/v1beta1	darts-cpu.yaml
katib/cmd/suggestion/nas/enas/v1beta1	enas-cpu.yaml

[i-chvets]

PR is created #105

[NohaIhab]

re-opening to add a test for the katib/cmd/metricscollector/v1beta1/tfevent-metricscollector image