-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dag for local data eval #13155
Dag for local data eval #13155
Commits on Oct 28, 2024
-
hcl2template: add refstring type for generic refs
The hcl2template package contains references already, but these are linked to a particular type. This becomes problematic if we want to support cross-type references, so this commit adds a new abstraction: refString. A refString contains the component type, its type (if applicable), and its name, so that the combination of those points to a cty object that can be linked to a block in the configuration. Right now, only `var`, `local` and `data` are supported, but the type is extensible enough that anything else that fits this model such as sources can be supported in the future potentially.
Configuration menu - View commit details
-
Copy full SHA for a3984f1 - Browse repository at this point
Copy the full SHA a3984f1View commit details -
hcl2template: rename Datasource.Name to DSName
The `Name` function is used by the DAG library to determine which vertex is associated to a component, so the `Name` attribute/function needs to replicate the combination of type and name, so the vertex can be accurately fetched from the graph.
Configuration menu - View commit details
-
Copy full SHA for 1b367d4 - Browse repository at this point
Copy the full SHA 1b367d4View commit details -
hcl2template: split local evaluation functions
Since we are in the process of integrating a new way to orchestrate dependency management and evaluation for datsources and local variables, we need to split the current function that manages the evaluation of said local variables, so that recursion and the actual evaluation of a local variable are two separate functions. This will allow for evaluating a single variable once the dag is ready to be introduced.
Configuration menu - View commit details
-
Copy full SHA for f66483f - Browse repository at this point
Copy the full SHA f66483fView commit details -
hcl2template: add function to eval one datasource
As with variables, this commit introduces a new function whose purpose is evaluating the datasource directly, without looking at its dependencies, or recursively trying to execute them. Instead, we rely on the DAG to determine when it is safe to execute it, or if using the phased approach, the current logic will apply.
Configuration menu - View commit details
-
Copy full SHA for 150b5a4 - Browse repository at this point
Copy the full SHA 150b5a4View commit details -
hcl2template: use refString for local/data deps
When registering dependencies for datasources and locals, we now use refString. This allows for the functions that detect dependencies to not only be able to register the same types as dependencies, but instead generalises it to any type that refString supports, so data, local and var. This can then be leveraged for orchestrating evaluation of those components in a non-phased way (i.e. with a DAG for dependency management).
Configuration menu - View commit details
-
Copy full SHA for ef0e9c2 - Browse repository at this point
Copy the full SHA ef0e9c2View commit details -
The dag package is a port over from Terraform to Packer, changing what little there was to fit our current dependency ecosystem. Most of the changes are on the type of diagnostics returned, as Terraform has its own type for them, while we rely on hcl's Diagnostics. Other than that, the functionality is essentially equivalent, and the code was barely touched.
Configuration menu - View commit details
-
Copy full SHA for 901440b - Browse repository at this point
Copy the full SHA 901440bView commit details -
hcl2template: add DAG-based eval for local/data
As we have finished setting-up the codebase for it, this commit adds the logic that uses the internal DAG package, and is able to orchestrate evaluation of datasources and locals in a non-phased way. Instead, this code acts by first detecting the dependencies for those components, builds a graph from them, with edges representing the dependency links between them, and finally walking on the graph breadth-first to evaluate those components. This can act as a drop-in replacement for the current phased logic, but both should be supported until we are confident that the approach works, and that there are little to no bugs left to squash.
Configuration menu - View commit details
-
Copy full SHA for 79f31e0 - Browse repository at this point
Copy the full SHA 79f31e0View commit details -
hcl2template: intro and add UseSequential init opt
Following up on the DAG work, this commit adds a new option for initialisation that disables DAG on request. By default we are going to use the DAG approach, with an option to fallback to using the older algorithm for evaluation in case users end-up in an edge-case that prevents them from building a template.
Configuration menu - View commit details
-
Copy full SHA for 8ba1913 - Browse repository at this point
Copy the full SHA 8ba1913View commit details -
command: add use-sequential options for commands
For all the commands that call Initialise, we introduce a new flag: UseSequential. This disables DAG scheduling for evaluating datasources and locals as a fallback to the newly introduced DAG scheduling approach. `hcl2_upgrade` is a special case here, as the template is always JSON, there cannot be any datasource, so the DAG in this case becomes meaningless, and is not integrated in this code path.
Configuration menu - View commit details
-
Copy full SHA for 5b0f20d - Browse repository at this point
Copy the full SHA 5b0f20dView commit details -
internal/dag: adapt Validate to not check for Root
The implementation of the DAG as extracted from Terraform relied on a Root vertex being injected into the graph as the last node to visit. This is used as a sanity check for Terraform, but doesn't apply to our use-case for now, as we are always executing everything and have no need for this root node. Instead, we change how Validate operates so it does not error in case there is no valid root node for the graph, but enables us calling it to check for self-referencing edges, and circular dependencies.
Configuration menu - View commit details
-
Copy full SHA for 6eaf62d - Browse repository at this point
Copy the full SHA 6eaf62dView commit details -
hcl2template: rename
Name
->LocalName
for localLocal variables had an attribute called Name with the name of the local variable. However, when producing an error while walking the DAG of local/datasources, if an error is encountered during validation, the raw structure of the vertex was printed out, making the error message produced hard to understand. Therefore in order to clean it up, we rename the `Name` attribute for Local variables as `LocalName`, and introduce a `Name()` function for that block so that the complete name of the variable is clearly reported.
Configuration menu - View commit details
-
Copy full SHA for d87cd1e - Browse repository at this point
Copy the full SHA d87cd1eView commit details -
hcl2template: don't use Walk for walking on DAG
Walk uses a reverse topological order to walk on the graph, doing that visit concurrently if possible. This is nice as we can speed-up execution of datasources and locals, however since the `Variables` map stored in the config, and the production of the context for it, are not meant to be used concurrently, this means that we end-up in cases where Packer crashes because of concurrent accesses to that map. So until we can change this behaviour, we will fallback to using the sequential visit algorithm for those vertexes, therefore limiting the risk of those conflicts.
Configuration menu - View commit details
-
Copy full SHA for 1467790 - Browse repository at this point
Copy the full SHA 1467790View commit details -
hcl2template: fix datasource dependency detection
When preparing the datasources to add into the DAG for evaluating the build prerequisites, we ended-up in a weird situation in which the datasources for each vertex pointed to the same one. This is because of the loop semantics of Go, where the same object is reused over and over again during each loop, so in the end every datasource vertex pointed to the same instance of a datasource block. To avoid this, we instead grab them through their reference, making the reference to the datasource purely local, and pointing to the actual datasource block, not the one scoped to the function.
Configuration menu - View commit details
-
Copy full SHA for 9189692 - Browse repository at this point
Copy the full SHA 9189692View commit details -
hcl2template: locals evaluation returns a variable
Evaluating local variables used to be directly written to the PackerConfig while each variable was created. This was somewhat of an issue with testing, as we have a bunch of tests that relied on `PackerConfig.Variables` being set only when we actually write something. This is not really a concern for normal use, just for testing, but to limit the number of changes to the tests in hcl2template, I opted to change how variables' values are retained, so that evaluating a single variable returns a Variable in addition to hcl.Diagnostics, so we can reify the approach and only create the map of variables if there's something evaluated.
Configuration menu - View commit details
-
Copy full SHA for eadcf63 - Browse repository at this point
Copy the full SHA eadcf63View commit details
Commits on Oct 29, 2024
-
packer_test: add base tests for DAG eval
Since we introduce the DAG with this series of commits, only on locals and data sources, we need to make sure that the behaviour is what we expect. Therefore, this commit adds a basic test with Packer build, and packer validate, to evaluate a template with both locals and data sources depending on one another. This is rejected with the sequential evaluation methods, as we process the different types one-by-one, whereas the DAG allows us to mix the order between the two, while still rejecting circular dependencies (and doing that before they even get evaluated), and self-references.
Configuration menu - View commit details
-
Copy full SHA for 024a204 - Browse repository at this point
Copy the full SHA 024a204View commit details -
internal/dag: remove unused code
Since the DAG package was lifted from Terraform, its contents are more than what we need for now, so this commit cleans-up the package to keep only the currently needed parts of code. If we need to support more in the future, we can revert this commit, or pickup the changes again from Terraform.
Configuration menu - View commit details
-
Copy full SHA for 539b85d - Browse repository at this point
Copy the full SHA 539b85dView commit details