In 0.12, modules can no longer be installed from local git repositories at relative paths #21107
Comments
|
Thanks for reporting this, @robrankin! I don't think this was intentionally supported in v0.11 and prior and was probably just a side-effect of an implementation detail that has changed now, but we'll see what we can do to restore the behavior nonetheless, since I can see that it would be useful. |
apparentlymart
changed the title
|
I am not sure if it is related but I am getting the same error with module "my_module" {
source = "git::ssh://[email protected]/private/repo-modules.git?ref=tag_v0.0.1//folder/sub_folder"
}am I missing something @apparentlymart ? |
|
@apparentlymart - We just ran into this same issue while working on upgrading our terraform repos. We use pinning of local repo hashes to support environment promotion. Do you think a fix for this might be coming soon? Thanks! |
|
|
I noticed similar issue. I have been using the module sources from git paths but when I switch the module source to local filesystem path, same module does not seem to work because of use of Also, when module source is git and terraform12 is initialized, modules get checked out and initialized in .terraform/modules/<module_name>/<full_path_to_module_directory_in_git> whereas for the localfilesystem, modules are simply symlinked under .terraform/modules/<module_name> --> module_directory_local_filesystem. Is this intentional? why is not there a copy of the module in the initialized directory instead? |
I have found the solution to the above issue. here is the correct way of sourcing private repositories in terraform assuming you have the right permissions to clone the repository |
|
source = "git::ssh://[email protected]:sre/vvv/legacy.git//aws_userdata?ref=master" |
|
For what it's worth, I've had success with Note the |
|
@salimane , @hunkjun , @maxmanders -- you guys are misunderstanding the problem that @robrankin posted, and I confirmed. Installing modules from remote git repos has always worked. Installing modules from local git repos, using relative paths, used to work prior to 0.12. This worked in 0.11: But no longer does in 0.12. It's super useful for local development without having to push commits to the remote before running init/plan/apply. |
|
Copy that, apologies for fuelling the confusion! |
|
Any updates regarding this issue? It's fairly tedious having to commit and push commits that don't work so I can find out that they don't work :) |
Basically changing things around and removing double slashes. AS per hashicorp/terraform#21107 (comment)
|
I just left a comment over in issue #25488 that I implemented a (forthcoming) fix for supporting |
There's an issue write-up in hashicorp/go-getter#268, as that's the library that provides the Git repo cloning support for Terraform. The string value of a Terraform module call's source parameter is parsed and handled by the The draft PR for it is hashicorp/go-getter#269 Assuming that PR (or something like it) gets merged, then getting the support for this feature in Terraform will require updating the dependency on go-getter to a version that includes the change. |
|
Thanks for capturing that context over in the I want to add a little more context here that is Terraform-specific: Terraform handles relative paths to modules already on disk (without the Implementing the handling of relative paths for git inside I'm not sure at this time what the best way forward is on this. In order to properly meet the use-cases described in that With all of that said, it seems like this might require some more thought to figure out what the appropriate separation of concerns is. Because |
@apparentlymart: Thanks for that additional context; that's quite helpful. I see what you mean about the module's context vs. $PWD -- my local testing setup was too simplistic to have that be an issue. In the work thus far on hashicorp/go-getter#269, there was definitely some friction against the current I def. agree that more thought is required. Thanks for the great feedback! |
@apparentlymart, Could I ask you to elaborate on that notion a little? The current |
|
Hi @salewski, Regarding the "is local filesystem access desirable?" question: I don't know if this actually matters in practice, but previously it seems like a non-Terraform If this did turn out to be true then one way we could potentially address it is to make the local file support opt-in, by whether this hypothetical new "base directory" field we've been talking about is set at all. Of course, whether that makes sense would depend on how the rest of that plays out, given the still-open question of whether this functionality can be the responsibility of the (On re-read I also see that you've suggested here that absolute local filesystem paths to git repositories have been working, in which case this would be less of a concern because it doesn't introduce any new access that wasn't present before. I didn't quite catch that on first read, so I was assuming that local directories were not possible at all before.) |
Thanks @apparentlymart. Makes sense; I'll keep it in mind.
Oh, in my comment above, I did not mean to imply that I knew absolute paths had been working previously. I did not find this GH issue until after I had done most of the work behind the current state of hashicorp/go-getter#269. My experience was that neither absolute nor relative filepaths worked with [0] only partially, as it turns out, at this point |
…nd relative
This series of changesets introduces a feature that allows the 'git::'
forcing token to be used on local file system paths to reference Git
repositories. Both absolute paths and relative paths are supported. For
example:
git::./some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::../../some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::/some/absolute/path/to/a/git-repo//some-subdir?ref=v4.5.6
Only filepaths that are prefixed with the 'git::' forcing token are
considered for processing.
Internally, go-getter transforms the provided string into a 'file://'
URI with an absolute filepath, with query string params and subdirectory
retained.
The rationale for using a 'file://' URI internally is that the Git clone
operation can already work with 'file://' URIs, and using them for this
feature allows us to leverage the existing go-getter URI-handling
machinery. That gets us support for query params (to clone a specific
git ref (tag, commit hash, ...)) "for free".
The rationale for using an absolute filepath (even when the provided
string is a relative filepath) is that (per RFC 1738 and RFC 8089) only
absolute filepaths are legitimate in 'file://' URIs. But more
importantly here, the Git clone operation only supports 'file://' URIs
with absolute paths.
Q: Why support this functionality at all?
Why not just require that a source location use an absolute path in a
'file://' URI explicitly if that's what is needed?
A: The primary reason is to allow support for relative filepaths to Git
repos.
There are use cases in which the absolute path cannot be known in
advance, but a relative path to a Git repo is known.
For example, when a Terraform project (or any Git-based project) uses
Git submodules, it will know the relative location of the Git
submodule repos, but cannot know the absolute path in advance because
it will vary based on where the "superproject" repo is
cloned. Nevertheless, those relative paths should be usable as
clonable Git repos, and this mechanism would allow for that.
Support for filepaths that are already absolute is provided mainly
for symmetry. It would be surprising for the feature to work with
relative file paths, but not for absolute filepaths.
For projects using Terraform, in particular, this feature (along with a
small change in the Terraform code to leverage it) enables the
non-fragile use of relative paths in a module "call" block, when
combined with Git submodules:
module "my_module" {
source = "git::../git-submodules/tf-modules/some-tf-module?ref=v0.1.0"
// ...
}
In the above example "superproject" Git repo (the one "calling" the
terraform module) knows the relative path to its own Git submodules
because they are embedded in a subdirectory beneath the top-level of the
"superproject" repo.
Two downstream Terraform issues that would require go-getter support for
this feature (or something like it) are at [0] and [1].
This first changeset in the series updates the README.md documentation
to note the new feature and provide examples.
[0] "Unable to use relative path to local Git module"
hashicorp/terraform#25488
[1] "In 0.12, modules can no longer be installed from local git repositories at relative paths"
hashicorp/terraform#21107
Design Notes
------------
In order for this feature to work, additional contextual information is
needed by the Git detector than can be provided using the existing
Detector API.
Internally, the Detector's Detect method does not pass along to the
Detector implementations all of the contextual information that it has
available. In particular, the forcing token and go-getter subdir
component are stripped out of the source string before invoking the
implementation's Detect method. In the particular case of the Git
detector, that means it cannot know that a 'git::' forcing token was
provided on an input string that otherwise looks like a file system
path. And /that/ means that it is not correct or safe for it to identify
any filepath string value as a Git repository.
Externally, callers (such as Terraform) already provide a value for the
'pwd' parameter of Detect, but it is not (necessarily) the location from
which a relative path in a 'git::' string should be resolved. In a
Terraform module (which may be in an arbitrary subdirectory from the
process current working directory), module "source" references that
contain relative paths must be interpreted relative to the location of
the module source file. Terraform has that information available, but in
the existing Detect API there is no way to convey it to go-getter.
Constraints
-----------
Additional Detector methods cannot be added without burdening all
existing detectors (both internal and in the wild) with the need to
support them.
Additional Detect method params cannot be added without breaking all
existing Detector implementations (internal, wild).
Additional parameters cannot be added to the Detect dispatching function
without affecting all callers.
Approach
--------
The goal is to provide the feature in a way that is as minimally
invasive as possible. But above all else it needs to avoid breaking
backward compatibility in any way.
Given that, the approach taken by this changeset series is to introduce
the concept of a "Contextual Detector". It is structured in the same way
as the current Detector framework, but works through a new CtxDetector
interface that is not constrained by the existing API.
The only callers affected by this change would be those that wish to take
advantage of the additional capabilities. And for those, the migration path
straight-forward because the new API is structured like the existing one.
In particular, this changeset series introduces four new elements:
1. CtxDetector interface
2. CtxDetect dispatching function
3. CtxDetect method on the CtxDetector interface
4. Full suite of CtxDetector implementations that are analogues of
the existing detectors (most of which (currently) just delegate
to the existing Detector implementations).
There is also a global 'ContextualDetectors' list that serves a function
analogous to the existing 'Detectors' list.
The go-getters library now has support for local file system paths to
Git repositories, specified with the 'git::' forcing token. The feature
works for both absolute and relative filepaths, and supports all the
usual go-getter goodies including '//' delimited subdirs and URI-style
query parameters.[0][1]
We incorporate that capability into Terraform, which allows users to
specify paths to locally present Git repositories from which to clone
other Terrform modules on which they are dependent. When coupled with
Git submodules, this creates a powerful way to manage Terraform modules
at specific versions without requiring those modules to be available on
the network (e.g., on GitHub):
module "my_module" {
source = "git::../git-submodules/tf-modules/some-tf-module?ref=v0.1.0"
// ...
}
From the perspective of Terraform, such Git repositories are "remote" in
the same way that repositories on GitHub are.
Note that within a Terraform module "call" block, the filepaths
specified are relative to the directory in which the *.tf file lives,
not relative to the current working directory of the Terraform
process. In order to support this feature, Terraform needs to supply
that contextual information to go-getter to allow relative filepath
resolution to work. In order to do so, we needed to switch over to using
go-getter's new "Contextual Detector" API. It works in the same basic
way as the traditional Detector API, but allows us to provide this
additional information.
In keeping with the "keep things simple" comment in the commit message
of 2b2ac1f, we are here maintaining our custom go-getter detectors
in two places. Only now each is called FooCtxDetector rather than
FooDetector. Nevertheless, all except the GitCtxDetector do little more
than "pass through" delegation to its analogous FooDetector counterpart.
Fixes hashicorp#25488
Fixes hashicorp#21107
[0] hashicorp/go-getter#268
[1] hashicorp/go-getter#269
…nd relative
This series of changesets introduces a feature that allows the 'git::'
forcing token to be used on local file system paths to reference Git
repositories. Both absolute paths and relative paths are supported. For
example:
git::./some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::../../some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::/some/absolute/path/to/a/git-repo//some-subdir?ref=v4.5.6
Only filepaths that are prefixed with the 'git::' forcing token are
considered for processing.
Internally, go-getter transforms the provided string into a 'file://'
URI with an absolute filepath, with query string params and subdirectory
retained.
The rationale for using a 'file://' URI internally is that the Git clone
operation can already work with 'file://' URIs, and using them for this
feature allows us to leverage the existing go-getter URI-handling
machinery. That gets us support for query params (to clone a specific
git ref (tag, commit hash, ...)) "for free".
The rationale for using an absolute filepath (even when the provided
string is a relative filepath) is that (per RFC 1738 and RFC 8089) only
absolute filepaths are legitimate in 'file://' URIs. But more
importantly here, the Git clone operation only supports 'file://' URIs
with absolute paths.
Q: Why support this functionality at all?
Why not just require that a source location use an absolute path in a
'file://' URI explicitly if that's what is needed?
A: The primary reason is to allow support for relative filepaths to Git
repos.
There are use cases in which the absolute path cannot be known in
advance, but a relative path to a Git repo is known.
For example, when a Terraform project (or any Git-based project) uses
Git submodules, it will know the relative location of the Git
submodule repos, but cannot know the absolute path in advance because
it will vary based on where the "superproject" repo is
cloned. Nevertheless, those relative paths should be usable as
clonable Git repos, and this mechanism would allow for that.
Support for filepaths that are already absolute is provided mainly
for symmetry. It would be surprising for the feature to work with
relative file paths, but not for absolute filepaths.
For projects using Terraform, in particular, this feature (along with a
small change in the Terraform code to leverage it) enables the
non-fragile use of relative paths in a module "call" block, when
combined with Git submodules:
module "my_module" {
source = "git::../git-submodules/tf-modules/some-tf-module?ref=v0.1.0"
// ...
}
In the above example "superproject" Git repo (the one "calling" the
terraform module) knows the relative path to its own Git submodules
because they are embedded in a subdirectory beneath the top-level of the
"superproject" repo.
Two downstream Terraform issues that would require go-getter support for
this feature (or something like it) are at [0] and [1].
This first changeset in the series updates the README.md documentation
to note the new feature and provide examples.
[0] "Unable to use relative path to local Git module"
hashicorp/terraform#25488
[1] "In 0.12, modules can no longer be installed from local git repositories at relative paths"
hashicorp/terraform#21107
Design Notes
------------
In order for this feature to work, additional contextual information is
needed by the Git detector than can be provided using the existing
Detector API.
Internally, the Detector's Detect method does not pass along to the
Detector implementations all of the contextual information that it has
available. In particular, the forcing token and go-getter subdir
component are stripped out of the source string before invoking the
implementation's Detect method. In the particular case of the Git
detector, that means it cannot know that a 'git::' forcing token was
provided on an input string that otherwise looks like a file system
path. And /that/ means that it is not correct or safe for it to identify
any filepath string value as a Git repository.
Externally, callers (such as Terraform) already provide a value for the
'pwd' parameter of Detect, but it is not (necessarily) the location from
which a relative path in a 'git::' string should be resolved. In a
Terraform module (which may be in an arbitrary subdirectory from the
process current working directory), module "source" references that
contain relative paths must be interpreted relative to the location of
the module source file. Terraform has that information available, but in
the existing Detect API there is no way to convey it to go-getter.
Constraints
-----------
Additional Detector methods cannot be added without burdening all
existing detectors (both internal and in the wild) with the need to
support them.
Additional Detect method params cannot be added without breaking all
existing Detector implementations (internal, wild).
Additional parameters cannot be added to the Detect dispatching function
without affecting all callers.
Approach
--------
The goal is to provide the feature in a way that is as minimally
invasive as possible. But above all else it needs to avoid breaking
backward compatibility in any way.
Given that, the approach taken by this changeset series is to introduce
the concept of a "Contextual Detector". It is structured in the same way
as the current Detector framework, but works through a new CtxDetector
interface that is not constrained by the existing API.
The only callers affected by this change would be those that wish to take
advantage of the additional capabilities. And for those, the migration path
straight-forward because the new API is structured like the existing one.
In particular, this changeset series introduces four new elements:
1. CtxDetector interface
2. CtxDetect dispatching function
3. CtxDetect method on the CtxDetector interface
4. Full suite of CtxDetector implementations that are analogues of
the existing detectors (most of which (currently) just delegate
to the existing Detector implementations).
There is also a global 'ContextualDetectors' list that serves a function
analogous to the existing 'Detectors' list.
…nd relative
This series of changesets introduces a feature that allows the 'git::'
forcing token to be used on local file system paths to reference Git
repositories. Both absolute paths and relative paths are supported. For
example:
git::./some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::../../some/relative/path/to/a/git-repo//some-subdir?ref=v1.2.3
or:
git::/some/absolute/path/to/a/git-repo//some-subdir?ref=v4.5.6
Only filepaths that are prefixed with the 'git::' forcing token are
considered for processing.
Internally, go-getter transforms the provided string into a 'file://'
URI with an absolute filepath, with query string params and subdirectory
retained.
The rationale for using a 'file://' URI internally is that the Git clone
operation can already work with 'file://' URIs, and using them for this
feature allows us to leverage the existing go-getter URI-handling
machinery. That gets us support for query params (to clone a specific
git ref (tag, commit hash, ...)) "for free".
The rationale for using an absolute filepath (even when the provided
string is a relative filepath) is that (per RFC 1738 and RFC 8089) only
absolute filepaths are legitimate in 'file://' URIs. But more
importantly here, the Git clone operation only supports 'file://' URIs
with absolute paths.
Q: Why support this functionality at all?
Why not just require that a source location use an absolute path in a
'file://' URI explicitly if that's what is needed?
A: The primary reason is to allow support for relative filepaths to Git
repos.
There are use cases in which the absolute path cannot be known in
advance, but a relative path to a Git repo is known.
For example, when a Terraform project (or any Git-based project) uses
Git submodules, it will know the relative location of the Git
submodule repos, but cannot know the absolute path in advance because
it will vary based on where the "superproject" repo is
cloned. Nevertheless, those relative paths should be usable as
clonable Git repos, and this mechanism would allow for that.
Support for filepaths that are already absolute is provided mainly
for symmetry. It would be surprising for the feature to work with
relative file paths, but not for absolute filepaths.
For projects using Terraform, in particular, this feature (along with a
small change in the Terraform code to leverage it) enables the
non-fragile use of relative paths in a module "call" block, when
combined with Git submodules:
module "my_module" {
source = "git::../git-submodules/tf-modules/some-tf-module?ref=v0.1.0"
// ...
}
In the above example "superproject" Git repo (the one "calling" the
terraform module) knows the relative path to its own Git submodules
because they are embedded in a subdirectory beneath the top-level of the
"superproject" repo.
Two downstream Terraform issues that would require go-getter support for
this feature (or something like it) are at [0] and [1].
This first changeset in the series updates the README.md documentation
to note the new feature and provide examples.
[0] "Unable to use relative path to local Git module"
hashicorp/terraform#25488
[1] "In 0.12, modules can no longer be installed from local git repositories at relative paths"
hashicorp/terraform#21107
Design Notes
------------
In order for this feature to work, additional contextual information is
needed by the Git detector than can be provided using the existing
Detector API.
Internally, the Detector's Detect method does not pass along to the
Detector implementations all of the contextual information that it has
available. In particular, the forcing token and go-getter subdir
component are stripped out of the source string before invoking the
implementation's Detect method. In the particular case of the Git
detector, that means it cannot know that a 'git::' forcing token was
provided on an input string that otherwise looks like a file system
path. And /that/ means that it is not correct or safe for it to identify
any filepath string value as a Git repository.
Externally, callers (such as Terraform) already provide a value for the
'pwd' parameter of Detect, but it is not (necessarily) the location from
which a relative path in a 'git::' string should be resolved. In a
Terraform module (which may be in an arbitrary subdirectory from the
process current working directory), module "source" references that
contain relative paths must be interpreted relative to the location of
the module source file. Terraform has that information available, but in
the existing Detect API there is no way to convey it to go-getter.
Constraints
-----------
Additional Detector methods cannot be added without burdening all
existing detectors (both internal and in the wild) with the need to
support them.
Additional Detect method params cannot be added without breaking all
existing Detector implementations (internal, wild).
Additional parameters cannot be added to the Detect dispatching function
without affecting all callers.
Approach
--------
The goal is to provide the feature in a way that is as minimally
invasive as possible. But above all else it needs to avoid breaking
backward compatibility in any way.
Given that, the approach taken by this changeset series is to introduce
the concept of a "Contextual Detector". It is structured in the same way
as the current Detector framework, but works through a new CtxDetector
interface that is not constrained by the existing API.
The only callers affected by this change would be those that wish to take
advantage of the additional capabilities. And for those, the migration path
straight-forward because the new API is structured like the existing one.
In particular, this changeset series introduces four new elements:
1. CtxDetector interface
2. CtxDetect dispatching function
3. CtxDetect method on the CtxDetector interface
4. Full suite of CtxDetector implementations that are analogues of
the existing detectors (most of which (currently) just delegate
to the existing Detector implementations).
There is also a global 'ContextualDetectors' list that serves a function
analogous to the existing 'Detectors' list.
Signed-off-by: Alan D. Salewski <[email protected]>
Terraform Version
Have reproduced this with beta1, beta2, and a dev compiled version from master today.
Terraform Configuration Files
Debug Output
Doesn't appear to be extremely useful, except perhaps this line:
Expected Behavior
terraform initshould succeedActual Behavior
terraform initfails as follows:0.12.0-beta1
0.12.0-beta2 and 0.12.0-dev
Steps to Reproduce
terraform initAdditional Context
These local filesystem git references work in 0.11.x
The text was updated successfully, but these errors were encountered: