Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dag for local data eval #13155

Merged
merged 16 commits into from
Oct 29, 2024
Merged

Dag for local data eval #13155

merged 16 commits into from
Oct 29, 2024

Commits on Oct 28, 2024

  1. hcl2template: add refstring type for generic refs

    The hcl2template package contains references already, but these are
    linked to a particular type.
    This becomes problematic if we want to support cross-type references, so
    this commit adds a new abstraction: refString.
    
    A refString contains the component type, its type (if applicable), and
    its name, so that the combination of those points to a cty object that
    can be linked to a block in the configuration.
    
    Right now, only `var`, `local` and `data` are supported, but the type is
    extensible enough that anything else that fits this model such as
    sources can be supported in the future potentially.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    a3984f1 View commit details
    Browse the repository at this point in the history
  2. hcl2template: rename Datasource.Name to DSName

    The `Name` function is used by the DAG library to determine which vertex
    is associated to a component, so the `Name` attribute/function needs to
    replicate the combination of type and name, so the vertex can be
    accurately fetched from the graph.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    1b367d4 View commit details
    Browse the repository at this point in the history
  3. hcl2template: split local evaluation functions

    Since we are in the process of integrating a new way to orchestrate
    dependency management and evaluation for datsources and local variables,
    we need to split the current function that manages the evaluation of
    said local variables, so that recursion and the actual evaluation of a
    local variable are two separate functions.
    
    This will allow for evaluating a single variable once the dag is ready
    to be introduced.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    f66483f View commit details
    Browse the repository at this point in the history
  4. hcl2template: add function to eval one datasource

    As with variables, this commit introduces a new function whose purpose
    is evaluating the datasource directly, without looking at its
    dependencies, or recursively trying to execute them.
    
    Instead, we rely on the DAG to determine when it is safe to execute it,
    or if using the phased approach, the current logic will apply.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    150b5a4 View commit details
    Browse the repository at this point in the history
  5. hcl2template: use refString for local/data deps

    When registering dependencies for datasources and locals, we now use
    refString.
    
    This allows for the functions that detect dependencies to not only be
    able to register the same types as dependencies, but instead generalises
    it to any type that refString supports, so data, local and var.
    
    This can then be leveraged for orchestrating evaluation of those
    components in a non-phased way (i.e. with a DAG for dependency
    management).
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    ef0e9c2 View commit details
    Browse the repository at this point in the history
  6. internal: add dag package

    The dag package is a port over from Terraform to Packer, changing what
    little there was to fit our current dependency ecosystem.
    Most of the changes are on the type of diagnostics returned, as
    Terraform has its own type for them, while we rely on hcl's Diagnostics.
    
    Other than that, the functionality is essentially equivalent, and the
    code was barely touched.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    901440b View commit details
    Browse the repository at this point in the history
  7. hcl2template: add DAG-based eval for local/data

    As we have finished setting-up the codebase for it, this commit adds the
    logic that uses the internal DAG package, and is able to orchestrate
    evaluation of datasources and locals in a non-phased way.
    
    Instead, this code acts by first detecting the dependencies for those
    components, builds a graph from them, with edges representing the
    dependency links between them, and finally walking on the graph
    breadth-first to evaluate those components.
    
    This can act as a drop-in replacement for the current phased logic, but
    both should be supported until we are confident that the approach works,
    and that there are little to no bugs left to squash.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    79f31e0 View commit details
    Browse the repository at this point in the history
  8. hcl2template: intro and add UseSequential init opt

    Following up on the DAG work, this commit adds a new option for
    initialisation that disables DAG on request.
    
    By default we are going to use the DAG approach, with an option to
    fallback to using the older algorithm for evaluation in case users
    end-up in an edge-case that prevents them from building a template.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    8ba1913 View commit details
    Browse the repository at this point in the history
  9. command: add use-sequential options for commands

    For all the commands that call Initialise, we introduce a new flag:
    UseSequential.
    
    This disables DAG scheduling for evaluating datasources and locals as a
    fallback to the newly introduced DAG scheduling approach.
    
    `hcl2_upgrade` is a special case here, as the template is always JSON,
    there cannot be any datasource, so the DAG in this case becomes
    meaningless, and is not integrated in this code path.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    5b0f20d View commit details
    Browse the repository at this point in the history
  10. internal/dag: adapt Validate to not check for Root

    The implementation of the DAG as extracted from Terraform relied on a
    Root vertex being injected into the graph as the last node to visit.
    
    This is used as a sanity check for Terraform, but doesn't apply to our
    use-case for now, as we are always executing everything and have no need
    for this root node.
    
    Instead, we change how Validate operates so it does not error in case
    there is no valid root node for the graph, but enables us calling it to
    check for self-referencing edges, and circular dependencies.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    6eaf62d View commit details
    Browse the repository at this point in the history
  11. hcl2template: rename Name->LocalName for local

    Local variables had an attribute called Name with the name of the local
    variable.
    
    However, when producing an error while walking the DAG of
    local/datasources, if an error is encountered during validation, the raw
    structure of the vertex was printed out, making the error message
    produced hard to understand.
    
    Therefore in order to clean it up, we rename the `Name` attribute for
    Local variables as `LocalName`, and introduce a `Name()` function for
    that block so that the complete name of the variable is clearly
    reported.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    d87cd1e View commit details
    Browse the repository at this point in the history
  12. hcl2template: don't use Walk for walking on DAG

    Walk uses a reverse topological order to walk on the graph, doing that
    visit concurrently if possible.
    
    This is nice as we can speed-up execution of datasources and locals,
    however since the `Variables` map stored in the config, and the
    production of the context for it, are not meant to be used concurrently,
    this means that we end-up in cases where Packer crashes because of
    concurrent accesses to that map.
    
    So until we can change this behaviour, we will fallback to using the
    sequential visit algorithm for those vertexes, therefore limiting the
    risk of those conflicts.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    1467790 View commit details
    Browse the repository at this point in the history
  13. hcl2template: fix datasource dependency detection

    When preparing the datasources to add into the DAG for evaluating the
    build prerequisites, we ended-up in a weird situation in which the
    datasources for each vertex pointed to the same one.
    
    This is because of the loop semantics of Go, where the same object is
    reused over and over again during each loop, so in the end every
    datasource vertex pointed to the same instance of a datasource block.
    
    To avoid this, we instead grab them through their reference, making the
    reference to the datasource purely local, and pointing to the actual
    datasource block, not the one scoped to the function.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    9189692 View commit details
    Browse the repository at this point in the history
  14. hcl2template: locals evaluation returns a variable

    Evaluating local variables used to be directly written to the
    PackerConfig while each variable was created.
    
    This was somewhat of an issue with testing, as we have a bunch of tests
    that relied on `PackerConfig.Variables` being set only when we actually
    write something.
    
    This is not really a concern for normal use, just for testing, but to
    limit the number of changes to the tests in hcl2template, I opted to
    change how variables' values are retained, so that evaluating a single
    variable returns a Variable in addition to hcl.Diagnostics, so we can
    reify the approach and only create the map of variables if there's
    something evaluated.
    lbajolet-hashicorp committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    eadcf63 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. packer_test: add base tests for DAG eval

    Since we introduce the DAG with this series of commits, only on locals
    and data sources, we need to make sure that the behaviour is what we
    expect.
    
    Therefore, this commit adds a basic test with Packer build, and packer
    validate, to evaluate a template with both locals and data sources
    depending on one another.
    
    This is rejected with the sequential evaluation methods, as we process
    the different types one-by-one, whereas the DAG allows us to mix the
    order between the two, while still rejecting circular dependencies (and
    doing that before they even get evaluated), and self-references.
    lbajolet-hashicorp committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    024a204 View commit details
    Browse the repository at this point in the history
  2. internal/dag: remove unused code

    Since the DAG package was lifted from Terraform, its contents are more
    than what we need for now, so this commit cleans-up the package to keep
    only the currently needed parts of code.
    If we need to support more in the future, we can revert this commit, or
    pickup the changes again from Terraform.
    lbajolet-hashicorp committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    539b85d View commit details
    Browse the repository at this point in the history