Introduce optional "refspec" to git PipelineResource, refactor #2320

ad22 · 2020-04-01T22:47:47Z

Changes

The git pipeline resource does not allow for any use case beyond a basic ref fetch, and is especially problematic for servers that don't support direct SHA fetches, which means that
the user cannot check out specific commit hashes not pointed to by a named ref. The user
is also not able to check out other refs alongside (example, the tag history). Current implementation also has some unexpected behavior, such as checking out/reset --hard
on the master branch after git-init (see #2282), running git pull without depth when checkout fails and so on . The end result is that several users might end up creating their own custom containers for fetching and checking out what they need. This change makes the git resource much more extensible, via an optional refspec parameter.

Addresses #2282

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

This change should be fully backward compatible w.r.t to revision parameter
EXCEPT the ability to fetch git commit short hashes. This was only possible
because the original implementation did a full repository pull when git fetch 
for the short SHA (unsupported) failed

- Adds ability to checkout older commit hashes on a particular ref
  chain (when fetch via commit SHA is not enabled on the server)
- Adds ability to fetch several other refs (for example, refs/tags)
  alongside the main revision
- Provides resolution for #2282 - If the user needs to checkout a branch,
  they must specify the appropriate refspec and revision as the ref of the
  branch. There was no way to do this in the original implementation
- Fixes bug reported in #2282 by ensuring the revision is always checked
  out on a detached HEAD when specified w/o refspec
- Fixes bug - depth parameter is not passed to git submodule update, leading to
  unnecessary data transfer
- Fixes bug --recurse-submodules is appended to the fetch command even when
  the user specified submodule: "false", leading to unnecessary data transfer

This is my first time contributing to this repository, please let me know if I missed something. Thanks!

googlebot · 2020-04-01T22:47:51Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

tekton-robot · 2020-04-01T22:48:03Z

Hi @ad22. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ad22 · 2020-04-01T22:49:06Z

@googlebot I signed it!

googlebot · 2020-04-01T22:49:09Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

vdemeester · 2020-04-02T06:57:14Z

/kind feature
/cc @chmouel
/ok-to-test

tekton-robot · 2020-04-02T06:59:50Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

tekton-robot · 2020-04-02T16:38:19Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

tekton-robot · 2020-04-02T17:56:48Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

tekton-robot · 2020-04-02T19:07:49Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

ad22 · 2020-04-02T20:10:57Z

/test pull-tekton-pipeline-integration-tests

tekton-robot · 2020-04-03T05:59:49Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

ghost

Thanks for the PR! I think this looks good, I just left a few style comments re: commenting and around the code structure with relation to our existing tests.

ghost · 2020-04-03T15:00:38Z

pkg/git/git.go

+		checkoutParam = spec.Revision
+	}
+
+	fetchArgs = append(fetchArgs, "origin", "--update-head-ok")


I didn't recognize this flag and looked it up in the docs, which say:

By default git fetch refuses to update the head which corresponds to the current branch. This flag disables the check. This is purely for the internal use for git pull to communicate with git fetch, and unless you are implementing your own Porcelain you are not supposed to use it.

I'm not sure what Porcelain is in this context but it would be great to include a note here on what the flag does and why we need to use it given that it's mostly intended for internal-to-git usage.

I've added a comment w/ explanation. Required because git init checks out the master branch by default.

ghost · 2020-04-03T15:16:38Z

test/git_checkout_test.go

+	t.Parallel()
+
+	// Pair of revision, refspec
+	var pairs = []string{


The approach we take elsewhere in our testing is to build a table of test cases with descriptions, inputs, and expected outputs (if any).

For example, I'd approach this test as follows:

for _, tc := range []struct{ description string revision string refspec string }{{ description: "Fetch refs/tags/v0.1.0 alongside master and checkout the master branch", revision: "master", refspec: "refs/tags/v0.1.0:refs/tags/v0.1.0 refs/heads/master:refs/heads/master", }, { description: "Checkout specific revision from refs/pull/1009/head's commit chain", revision: ""968d5d37a61bfb85426c885dc1090c1cc4b33436"", refspec: "refs/pull/1009/head", }, { // ...and so on... }} { t.Run(tc.description, func(t *testing.T) { // test body }) }

The benefit is that this moves the comments into test names, which get logged on failure, and it adds a clean field name for each of the entries in your current pairs list. It would also match more closely what we're doing elsewhere, which is nice from a pure code-style perspective.

@sbwsg Thanks for the explanation. That's a great way to organize the test cases. I've updated the PR.

ghost · 2020-04-03T15:17:58Z

test/v1alpha1/git_checkout_test.go

+	t.Parallel()
+
+	// Pair of revision, refspec
+	var pairs = []string{


Same comment here as in the other test file: it would be great to see this turned into a table with field names.

tekton-robot · 2020-04-03T21:12:55Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

ghost · 2020-04-06T14:22:25Z

test/v1alpha1/git_checkout_test.go

+			knativetest.CleanupOnInterrupt(func() { tearDown(t, c, namespace) }, t.Logf)
+			defer tearDown(t, c, namespace)
+
+			t.Logf("Creating Git PipelineResource %s", gitSourceResourceName)


I think it makes sense to log in the failure case with the t.Fatalf but these kinds of status lines can make test logs quite noisy to trace through when looking at verbose output. I suggest removing the use of t.Logf unless absolutely necessary. What do you think?

I've removed the log entries in the latest squashed commit.

tekton-robot · 2020-04-06T14:22:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbwsg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbwsg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afrittoli · 2020-04-06T20:09:22Z

Thank you for your contribution. Please note that we do not auto-squash commits on merge, so I would ask you if you could squash your commits down to one or two.

afrittoli · 2020-04-06T20:13:19Z

pkg/apis/pipeline/v1alpha1/pipeline_resource_types.go

@@ -30,7 +30,7 @@ var (
 )

 const (
-	// PipelineResourceTypeGit indicates that this source is a GitHub repo.
+	// PipelineResourceTypeGit indicates that this source is a Git repo.


afrittoli · 2020-04-06T20:19:32Z

pkg/git/git.go

+	// The --force parameter tells git-fetch that its ok to update an existing HEAD in a
+	// non-fast-forward manner (though this cannot be possible on initial fetch, it can help
+	// when the refspec specifies the same destination twice)
+	fetchArgs = append(fetchArgs, "origin", "--update-head-ok", "--force")


Would it make sense to limit the use of --update-head-ok to the case where the revision is "master"?

I don't see the value in putting an additional conditional for that case. This is (guaranteed to be) the first fetch command run after git-init, so its implied that
a. it will always be the master branch, unless git-init changes behavior
b. we are not expecting any unintended updates to existing refs

afrittoli · 2020-04-06T20:23:01Z

cmd/git-init/main.go

 	flag.StringVar(&fetchSpec.Path, "path", "", "Path of directory under which Git repository will be copied")
 	flag.BoolVar(&fetchSpec.SSLVerify, "sslVerify", true, "Enable/Disable SSL verification in the git config")
-	flag.BoolVar(&submodules, "submodules", true, "Initialize and fetch Git submodules")
+	flag.BoolVar(&fetchSpec.Submodules, "submodules", true, "Initialize and fetch Git submodules")


This change does not seem to be needed for this patch, but I guess it makes sense anyways.

This is actually needed for the patch, to pass the submodule setting to git.go and prevent --recurse-submodules from being appended to the fetch command when
the user specified submodule: "false"

Add refspec support to git task corresponding to tektoncd/pipeline#2320 Add a new batch merge task (tektoncd/pipeline#508) that re-uses the git-init container and subsequently performs the requested batch merge mode on top of the checked out revision

vdemeester · 2020-04-17T09:20:13Z

@ad22 needs a rebase, sorry 🙏

The git pipeline resource does not allow for any use case beyond a basic ref fetch, and is especially problematic for servers that don't support direct SHA fetches, which means that the user cannot check out specific commit hashes not pointed to by a named ref. The user is also not able to check out other refs alongside (example, the tag history). Current implementation also has a few corner cases, such as checking out/reset --hard on the master branch after git-init, running git reset --hard when checkout fails and so on. This leads to inconsistent behavior (see tektoncd#2282). The end result is that several users might end up creating their own custom containers for fetching and checking out what they need. This change makes the git resource much more extensible, via an optional refspec parameter. This change should be fully backward compatible w.r.t to revision parameter EXCEPT the ability to fetch git commit short hashes (according to the e2e tests) - Adds ability to checkout older commit hashes on a particular ref chain (when fetch via commit SHA is not enabled on the server) - Adds ability to fetch several other refs (for example, refs/tags) alongside the main revision - Provides resolution for tektoncd#2282 - If the user needs to checkout a branch, they must specify the appropriate refspec and revision as the ref of the branch. There was no way to do this in the original implementation - Fixes bug reported in tektoncd#2282 by ensuring the revision is always checked out on a detached HEAD when specified w/o refspec - Fixes bug - depth parameter is not passed to git submodule update, leading to unnecessary data transfer - Fixes bug --recurse-submodules is appended to the fetch command even when the user specified submodule: "false", leading to unnecessary data transfer - Fixes bug tektoncd#1843 - git init ssh symlink is failing Addresses tektoncd#2282, tektoncd#1843

tekton-robot · 2020-04-17T15:54:44Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

ad22 · 2020-04-17T15:55:58Z

Rebased and added minor fix for #1843

tekton-robot · 2020-04-17T15:58:31Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/resource/v1alpha1/git/git_resource.go	84.4%	85.4%	1.0

ghost · 2020-04-17T16:52:33Z

/test pull-tekton-pipeline-build-tests

ghost · 2020-04-17T18:58:46Z

I'm going to merge this now and if there are any outstanding comments that need to be addressed we can tackle them in a follow-up. Thanks for the PR @ad22 !

/lgtm

Add refspec support to git task corresponding to tektoncd/pipeline#2320 Add a new batch merge task (tektoncd/pipeline#508) that re-uses the git-init container and subsequently performs the requested batch merge mode on top of the checked out revision

googlebot added the cla: no label Apr 1, 2020

tekton-robot requested review from a user and vdemeester April 1, 2020 22:47

tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 1, 2020

googlebot added cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit and removed cla: no labels Apr 1, 2020

ad22 force-pushed the master branch from c83ed5e to 178607d Compare April 2, 2020 03:51

tekton-robot requested a review from chmouel April 2, 2020 06:57

tekton-robot added kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 2, 2020

ghost mentioned this pull request Apr 3, 2020

Pin git-clone to the 0.11.0 version of git-init tektoncd/catalog#252

Merged

1 task

ghost suggested changes Apr 3, 2020

View reviewed changes

ghost approved these changes Apr 6, 2020

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 6, 2020

afrittoli reviewed Apr 6, 2020

View reviewed changes

This was referenced Apr 13, 2020

Add batch merge and refspec support to git task tektoncd/catalog#273

Merged

git-init ssh symlink is failing #1843

Closed

ghost mentioned this pull request Apr 16, 2020

Pipeline Resource of type git doesn't properly handle revisions that are tags #2425

Closed

vdemeester added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2020

ad22 force-pushed the master branch from a872186 to 059dee7 Compare April 17, 2020 15:51

ad22 force-pushed the master branch from 059dee7 to 5bc2045 Compare April 17, 2020 15:53

ghost removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2020

tekton-robot assigned ghost Apr 17, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 17, 2020

tekton-robot merged commit 8bc424d into tektoncd:master Apr 17, 2020

ghost mentioned this pull request Apr 27, 2020

Git Resources are always on "master" branch. #2282

Closed

AlanGreene mentioned this pull request Jun 22, 2020

Add support for specifying git revision on Import Resources form tektoncd/dashboard#1550

Closed

chhsia0 mentioned this pull request Jul 28, 2020

Git CLI doesn't work on Git resource that is an output resource of another task. #3027

Closed

ghost mentioned this pull request Feb 9, 2021

Policy for when to bump versions of Tasks that depend on images in tektoncd/pipeline? tektoncd/catalog#644

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce optional "refspec" to git PipelineResource, refactor #2320

Introduce optional "refspec" to git PipelineResource, refactor #2320

ad22 commented Apr 1, 2020 •

edited

Loading

googlebot commented Apr 1, 2020

tekton-robot commented Apr 1, 2020

ad22 commented Apr 1, 2020

googlebot commented Apr 1, 2020

vdemeester commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

ad22 commented Apr 2, 2020

tekton-robot commented Apr 3, 2020

ghost left a comment

ghost Apr 3, 2020

ad22 Apr 3, 2020

ghost Apr 3, 2020

ad22 Apr 3, 2020

ghost Apr 3, 2020

tekton-robot commented Apr 3, 2020

ghost Apr 6, 2020

ad22 Apr 6, 2020

tekton-robot commented Apr 6, 2020

afrittoli commented Apr 6, 2020

afrittoli Apr 6, 2020

afrittoli Apr 6, 2020

ad22 Apr 6, 2020

afrittoli Apr 6, 2020

ad22 Apr 6, 2020

vdemeester commented Apr 17, 2020

tekton-robot commented Apr 17, 2020

ad22 commented Apr 17, 2020

tekton-robot commented Apr 17, 2020

ghost commented Apr 17, 2020

ghost commented Apr 17, 2020

Introduce optional "refspec" to git PipelineResource, refactor #2320

Introduce optional "refspec" to git PipelineResource, refactor #2320

Conversation

ad22 commented Apr 1, 2020 • edited Loading

Changes

Submitter Checklist

Reviewer Notes

Release Notes

googlebot commented Apr 1, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

tekton-robot commented Apr 1, 2020

ad22 commented Apr 1, 2020

googlebot commented Apr 1, 2020

vdemeester commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

tekton-robot commented Apr 2, 2020

ad22 commented Apr 2, 2020

tekton-robot commented Apr 3, 2020

ghost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Apr 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Apr 6, 2020

afrittoli commented Apr 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester commented Apr 17, 2020

tekton-robot commented Apr 17, 2020

ad22 commented Apr 17, 2020

tekton-robot commented Apr 17, 2020

ghost commented Apr 17, 2020

ghost commented Apr 17, 2020

ad22 commented Apr 1, 2020 •

edited

Loading