Problem with passing the provider to the tests

[jbtk]

Hi!

I am trying to run e2e autoscaler tests on a cluster on GCE.

When I run:
kubetest2 gce -v 2 --repo-root --gcp-project --legacy-mode --build --up --down --test=ginkgo -- --focus-regex='[Feature:ClusterSizeAutoscalingScaleUp]|[Feature:ClusterSizeAutoscalingScaleDown]|[Feature:InitialResources]' --ginkgo-args="-v"

Most of the tests are skipped because it seems that the tester does not have information which provider was used to provision the cluster for the test. I get information:
[SKIPPED] Only supported for providers [gce gke] (not skeleton)

I tried looking for "skeleton" and I see it in only one place in kubetest2 repo - in the clusterloader2 which seems to be only used by the clusterloader tester so I am not sure why this is a problem with ginkgo - I am probably missing some call stack as in the code they do not seem to depend on each other.

Am I doing something wrong? Is it a known issue?

[jbtk]

Testing some more:

• the clusterloader2 is not relevant
• the provider param is missing in the ginkgo runner and since it is not passed the test_context.go uses skeleton as a fallback

[jbtk]

I have read the discussion in kubernetes/kubernetes#70194 and I am still not sure what is the up to date plan going forward, but it seems that the best option would be to make the tests that I am running provider agnostic and remove dependency on the provider. I will look into that, but I am still not sure whether it is expected for kubetest2 not to support passing provider or is it a missing functionality. Would be happy to learn that. I could contribute a fix here, but would like to make sure that it is indeed wanted.

[jbtk]

It seems that even after hacking (hardcoding locally to check whether it would work) and passing the provider the test will not run as they require other context args that are not populated (as defined in https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/test_context.go#L431). Would it make sense for deployer to prepare set of provider specific e2e test args that could be read from the rundir similarly to the kubeconfig?

[jbtk]

I found a workaround to pass the provider manually in the test args. Example:
kubetest2 gce -v 2 --repo-root --gcp-project --legacy-mode --run-id= --test=ginkgo -- --focus-regex='' --ginkgo-args="-v" --test-args='--provider=gce --gce-zone= --gce-project= --node-instance-group='

Not perfect, but works*

*not fully as it fails to find logs on master and nodes to dump to the artifacts, but the tests are running

[jbtk]

If I want the log dump to work I need to apply a change in the code (PR: #264) and pass additional env variable:
--env=KUBE_GCE_INSTANCE_PREFIX=

Without the PR passing the env variable would clear other variables and fail the test.

[jbtk]

The in-tree provider has been removed and running tests in legacy mode is not working any more. I will need to figure out how to run the tests again.

[jbtk]

Closing this issue for now, will open another one when/if I run into issues.