Speed up TestDiffConvergence #1860
Labels
area/testing
The internal testing setup of the bridge
kind/engineering
Work that is not visible to an external user
Comments
t0yv0
added a commit
that referenced
this issue
Apr 12, 2024
Fixes #1790 by building a rapid generator for schemas and associated values. Large-ish problem 1: I do not have it figured out how to test unknown values. TF literals as unknown values are forbidden and do not make sense. We might need a helper resource so that testing unknown values generates references to an output of the helper resource. This is logged for future work. Large-ish problem 2: iteration is pretty slow (x-proc). Normal n=100 rapid tests can take up to 10min. Could try batching so several resources are tried in one shot say 100 resources. Large-ish problem 3: I'm not sure if no-op Update and Create implementations are acceptable. There is something to testing Computed attributes where provider has to set values. Possibly Update also needs to set values? Possibly not. Small problems: - [x] Using TF JSON syntax didn't handle null/empty correctly; that is now discarded, using actual HCL syntax - [x] TF representations are difficult to visualize in failing tests and difficult to assert against - [x] Lots of lost-in-translation papercuts possible between representations (cty.Value, resource.PropertyValue, tftypes.Value) - [x] this requires a change to providertest to abstract from testing.T so we can pass rapid.T - [x] it's very hard to disable annoying TF logging, using env vars for now We are starting to find bugs and discrepancies from this work: - #1856 panic corner-case - #1852 need to InternalValidate - #1828 Future work: - #1856 - #1857 - #1858 - #1859 - #1860 - #1861 - #1862 - #1863 - #1864 - #1865 - #1866 - #1867
iwahbe
added
kind/engineering
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With #1801 rapid tests are introduced that drive Pulumi and TF CLI but one iteration takes about 10 seconds which is
slow. There is no easy way to make rapid tests parallel, however it is possible to cheat by rebuilding a generator so
that it samples from 0-100 resource instances a shot, and reduces the maps to a 1-element map once an error is found.
This way the cost of running CLIs under the hood will be amortized as more combinations are tried at once.
The text was updated successfully, but these errors were encountered: