Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build a rapid Generator for SDKV2 Schema/Value pairs #1790

Closed
Tracked by #1796
t0yv0 opened this issue Mar 26, 2024 · 0 comments · Fixed by #1801
Closed
Tracked by #1796

Build a rapid Generator for SDKV2 Schema/Value pairs #1790

t0yv0 opened this issue Mar 26, 2024 · 0 comments · Fixed by #1801
Assignees
Labels
kind/engineering Work that is not visible to an external user resolution/fixed This issue was fixed
Milestone

Comments

@t0yv0
Copy link
Member

t0yv0 commented Mar 26, 2024

To facilitate testing across the space of possible SDKv2 schemas, build a generator that creates tuples (s, v) of a schema s and a TF value that conforms to the schema v: s.

There are some important variations to cover

  • every possible combination of schema, up to depth 3, including maps, sets and objects
  • within the schema, every possible combination of Attribute, Block, and nesting modes (SingleNested, ListNested, etc)
  • every possible combination of attribute kind, Computed, Optional, Computed+Optional, Required
  • sets, including several distinct custom set hashes
  • objects need to have up to 2 attributes so Set[Obj] combination cover variations of attribute kind
  • when generating values, ensure to cover null, unknown, and empty values and 2 regular values, say for strings [null, unknown, "", "a", "b" ]
@t0yv0 t0yv0 added the kind/engineering Work that is not visible to an external user label Mar 26, 2024
@t0yv0 t0yv0 mentioned this issue Mar 26, 2024
8 tasks
t0yv0 added a commit that referenced this issue Apr 12, 2024
Fixes #1790 by
building a rapid generator for schemas and associated values.

Large-ish problem 1: I do not have it figured out how to test unknown
values. TF literals as unknown values are forbidden and do not make
sense. We might need a helper resource so that testing unknown values
generates references to an output of the helper resource. This is logged
for future work.

Large-ish problem 2: iteration is pretty slow (x-proc). Normal n=100
rapid tests can take up to 10min. Could try batching so several
resources are tried in one shot say 100 resources.

Large-ish problem 3: I'm not sure if no-op Update and Create
implementations are acceptable. There is something to testing Computed
attributes where provider has to set values. Possibly Update also needs
to set values? Possibly not.

Small problems:

- [x] Using TF JSON syntax didn't handle null/empty correctly; that is
now discarded, using actual HCL syntax
- [x] TF representations are difficult to visualize in failing tests and
difficult to assert against
- [x] Lots of lost-in-translation papercuts possible between
representations (cty.Value, resource.PropertyValue, tftypes.Value)
- [x] this requires a change to providertest to abstract from testing.T
so we can pass rapid.T
- [x] it's very hard to disable annoying TF logging, using env vars for
now

We are starting to find bugs and discrepancies from this work:

- #1856 panic
corner-case
- #1852 need to
InternalValidate
- #1828

Future work:

- #1856 
- #1857 
- #1858 
- #1859 
- #1860 
- #1861 
- #1862 
- #1863 
- #1864 
- #1865 
- #1866 
- #1867
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label Apr 12, 2024
@t0yv0 t0yv0 added this to the 0.103 milestone Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/engineering Work that is not visible to an external user resolution/fixed This issue was fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants