Cross-test delete-replace plans #1865
Labels
area/testing
The internal testing setup of the bridge
kind/engineering
Work that is not visible to an external user
Comments
t0yv0
added a commit
that referenced
this issue
Apr 12, 2024
Fixes #1790 by building a rapid generator for schemas and associated values. Large-ish problem 1: I do not have it figured out how to test unknown values. TF literals as unknown values are forbidden and do not make sense. We might need a helper resource so that testing unknown values generates references to an output of the helper resource. This is logged for future work. Large-ish problem 2: iteration is pretty slow (x-proc). Normal n=100 rapid tests can take up to 10min. Could try batching so several resources are tried in one shot say 100 resources. Large-ish problem 3: I'm not sure if no-op Update and Create implementations are acceptable. There is something to testing Computed attributes where provider has to set values. Possibly Update also needs to set values? Possibly not. Small problems: - [x] Using TF JSON syntax didn't handle null/empty correctly; that is now discarded, using actual HCL syntax - [x] TF representations are difficult to visualize in failing tests and difficult to assert against - [x] Lots of lost-in-translation papercuts possible between representations (cty.Value, resource.PropertyValue, tftypes.Value) - [x] this requires a change to providertest to abstract from testing.T so we can pass rapid.T - [x] it's very hard to disable annoying TF logging, using env vars for now We are starting to find bugs and discrepancies from this work: - #1856 panic corner-case - #1852 need to InternalValidate - #1828 Future work: - #1856 - #1857 - #1858 - #1859 - #1860 - #1861 - #1862 - #1863 - #1864 - #1865 - #1866 - #1867
iwahbe
added
kind/engineering
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With cross-testing introduced in #1790 it might be possible for the randomly generated state transitions to pick a Delete-Replace plan, however this has not been completed yet. We need to:
The text was updated successfully, but these errors were encountered: