Skip to content

Commit

Permalink
git: add push.refspec to push using a refspec
Browse files Browse the repository at this point in the history 
Add `.spec.git.push.refspec` to allow specifying a refspec to be used
for performing a push operation. It takes precedence over the
`spec.git.push.branch` field.

Signed-off-by: Sanskar Jaiswal <[email protected]>
  • Loading branch information
@aryan9600
aryan9600 committed Jun 15, 2023
1 parent 741227f commit 312426b
Show file tree
Hide file tree
Showing 6 changed files with 242 additions and 75 deletions.
11 changes: 9 additions & 2 deletions api/v1beta1/git.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,13 @@ type PushSpec struct {
// Branch specifies that commits should be pushed to the branch
// named. The branch is created using `.spec.checkout.branch` as the
// starting point, if it doesn't already exist.
// +required
Branch string `json:"branch"`
// +optional
Branch string `json:"branch,omitempty"`

// Refspec specifies the Git Refspec to use for a push operation.
// It takes precedence over Branch, i.e. Branch is ignored
// if Refspec is non empty. For more details about Git Refspecs, see:
// https://git-scm.com/book/en/v2/Git-Internals-The-Refspec
// +optional
Refspec string `json:"refspec,omitempty"`
}
8 changes: 6 additions & 2 deletions config/crd/bases/image.toolkit.fluxcd.io_imageupdateautomations.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,12 @@ spec:
to the branch named. The branch is created using `.spec.checkout.branch`
as the starting point, if it doesn't already exist.
type: string
required:
- branch
refspec:
description: 'Refspec specifies the Git Refspec to use for
a push operation. It takes precedence over Branch, i.e.
Branch is ignored if Refspec is non empty. For more details
about Git Refspecs, see: https://git-scm.com/book/en/v2/Git-Internals-The-Refspec'
type: string
type: object
required:
- commit
Expand Down
16 changes: 16 additions & 0 deletions docs/api/image-automation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -638,11 +638,27 @@ string
</em>
</td>
<td>
<em>(Optional)</em>
<p>Branch specifies that commits should be pushed to the branch
named. The branch is created using <code>.spec.checkout.branch</code> as the
starting point, if it doesn&rsquo;t already exist.</p>
</td>
</tr>
<tr>
<td>
<code>refspec</code><br>
<em>
string
</em>
</td>
<td>
<em>(Optional)</em>
<p>Refspec specifies the Git Refspec to use for a push operation.
It takes precedence over Branch, i.e. Branch is ignored
if Refspec is non empty. For more details about Git Refspecs, see:
<a href="https://git-scm.com/book/en/v2/Git-Internals-The-Refspec">https://git-scm.com/book/en/v2/Git-Internals-The-Refspec</a></p>
</td>
</tr>
</tbody>
</table>
</div>
Expand Down
34 changes: 31 additions & 3 deletions docs/spec/v1beta1/imageupdateautomations.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -406,8 +406,15 @@ type PushSpec struct {
// Branch specifies that commits should be pushed to the branch
// named. The branch is created using `.spec.checkout.branch` as the
// starting point, if it doesn't already exist.
// +required
Branch string `json:"branch"`
// +optional
Branch string `json:"branch,omitempty"`

// Refspec specifies the Git Refspec to use for a push operation.
// It takes precedence over Branch, i.e. Branch is ignored
// if Refspec is non empty. For more details about Git Refspecs, see:
// https://git-scm.com/book/en/v2/Git-Internals-The-Refspec
// +optional
Refspec string `json:"refspec,omitempty"`
}
```

Expand All @@ -416,7 +423,11 @@ pushed to the same branch at the origin. If `.spec.git.checkout` is not present,
to the branch given in the `GitRepository` referenced by `.spec.sourceRef`. If none of these yield a
branch name, the automation will fail.

When `push` is present, the `branch` field specifies a branch to push to at the origin. The branch
If `push.refspec` is present, the refspec specified is used to perform the push operation.
An example of a valid refspec is `refs/heads/branch:refs/heads/branch`. This allows users to push to an
arbitary destination reference.

If `push.branch` is present, the specified branch is pushed to at the origin. The branch
will be created locally if it does not already exist, starting from the checkout branch. If it does
already exist, it will be overwritten with the cloned version plus the changes made by the
controller. Alternatively, force push can be disabled by starting the controller with `--feature-gates=GitForcePushBranch=false`,
Expand All @@ -425,6 +436,8 @@ Note that without force push in push branches, if the target branch is stale, th
be able to conclude the operation and will consistently fail until the branch is either deleted or
refreshed.

**Note:** If `push.refspec` is specified, then `push.branch` is ignored.

In the following snippet, updates will be pushed as commits to the branch `auto`, and when that
branch does not exist at the origin, it will be created locally starting from the branch `main`, and
pushed:
Expand All @@ -439,6 +452,21 @@ spec:
branch: auto
```
In the following snippet, updates and commits will be made on the `main` branch locally
and then pushed to the `qa/main` branch. Note that, the `auto` branch specified in
`spec.git.push.branch` is ignored.

```yaml
spec:
git:
checkout:
ref:
branch: main
push:
branch: auto
refspec: refs/heads/main:refs/heads/qa/main
```

## Update strategy

The `.spec.update` field specifies how to carry out updates on the git repository. There is one
Expand Down
152 changes: 95 additions & 57 deletions internal/controller/imageupdateautomation_controller.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -214,30 +214,15 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
}

// validate the git spec and default any values needed later, before proceeding
var ref *sourcev1.GitRepositoryRef
var checkoutRef *sourcev1.GitRepositoryRef
if gitSpec.Checkout != nil {
ref = &gitSpec.Checkout.Reference
tracelog.Info("using git repository ref from .spec.git.checkout", "ref", ref)
checkoutRef = &gitSpec.Checkout.Reference
tracelog.Info("using git repository ref from .spec.git.checkout", "ref", checkoutRef)
} else if r := origin.Spec.Reference; r != nil {
ref = r
tracelog.Info("using git repository ref from GitRepository spec", "ref", ref)
checkoutRef = r
tracelog.Info("using git repository ref from GitRepository spec", "ref", checkoutRef)
} // else remain as `nil` and git.DefaultBranch will be used.

var pushBranch string
if gitSpec.Push != nil {
pushBranch = gitSpec.Push.Branch
tracelog.Info("using push branch from .spec.push.branch", "branch", pushBranch)
} else {
// Here's where it gets constrained. If there's no push branch
// given, then the checkout ref must include a branch, and
// that can be used.
if ref == nil || ref.Branch == "" {
return failWithError(fmt.Errorf("Push branch not given explicitly, and cannot be inferred from .spec.git.checkout.ref or GitRepository .spec.ref"))
}
pushBranch = ref.Branch
tracelog.Info("using push branch from $ref.branch", "branch", pushBranch)
}

tmp, err := os.MkdirTemp("", fmt.Sprintf("%s-%s", originName.Namespace, originName.Name))
if err != nil {
return failWithError(err)
Expand All @@ -248,42 +233,44 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
}
}()

debuglog.Info("attempting to clone git repository", "gitrepository", originName, "ref", ref, "working", tmp)

authOpts, err := r.getAuthOpts(ctx, &origin)
if err != nil {
return failWithError(err)
}

clientOpts := []gogit.ClientOption{gogit.WithDiskStorage()}
if authOpts.Transport == git.HTTP {
clientOpts = append(clientOpts, gogit.WithInsecureCredentialsOverHTTP())
var pushBranch string
var switchBranch bool
if gitSpec.Push != nil {
// We only need to switch branches when a branch has been specified in
// the push spec and a refspec has not. Furthermore, the branch needs to
// be different than the one in the checkout ref.
if gitSpec.Push.Branch != "" && gitSpec.Push.Branch != checkoutRef.Branch && gitSpec.Push.Refspec == "" {
pushBranch = gitSpec.Push.Branch
switchBranch = true
tracelog.Info("using push branch from .spec.push.branch", "branch", pushBranch)
}
} else {
// Here's where it gets constrained. If there's no push branch
// given, then the checkout ref must include a branch, and
// that can be used.
if checkoutRef == nil || checkoutRef.Branch == "" {
return failWithError(
fmt.Errorf("Push spec not provided, and cannot be inferred from .spec.git.checkout.ref or GitRepository .spec.ref"),
)
}
pushBranch = checkoutRef.Branch
tracelog.Info("using push branch from $ref.branch", "branch", pushBranch)
}

// If the push branch is different from the checkout ref, we need to
// have all the references downloaded at clone time, to ensure that
// SwitchBranch will have access to the target branch state. fluxcd/flux2#3384
//
// To always overwrite the push branch, the feature gate
// GitAllBranchReferences can be set to false, which will cause
// the SwitchBranch operation to ignore the remote branch state.
allReferences := r.features[features.GitAllBranchReferences]
if pushBranch != ref.Branch {
clientOpts = append(clientOpts, gogit.WithSingleBranch(!allReferences))
}
debuglog.Info("attempting to clone git repository", "gitrepository", originName, "ref", checkoutRef, "working", tmp)

gitClient, err := gogit.NewClient(tmp, authOpts, clientOpts...)
gitClient, err := r.constructGitClient(ctx, &origin, tmp, switchBranch)
if err != nil {
return failWithError(err)
}
defer gitClient.Close()

opts := repository.CloneConfig{}
if ref != nil {
opts.Tag = ref.Tag
opts.SemVer = ref.SemVer
opts.Commit = ref.Commit
opts.Branch = ref.Branch
if checkoutRef != nil {
opts.Tag = checkoutRef.Tag
opts.SemVer = checkoutRef.SemVer
opts.Commit = checkoutRef.Commit
opts.Branch = checkoutRef.Branch
}

if enabled, _ := r.features[features.GitShallowClone]; enabled {
Expand All @@ -297,9 +284,9 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
return failWithError(err)
}

// When there's a push spec, the pushed-to branch is where commits
// When there's a push branch specified, the pushed-to branch is where commits
// shall be made
if gitSpec.Push != nil && !(ref != nil && ref.Branch == pushBranch) {
if switchBranch {
// Use the git operations timeout for the repo.
fetchCtx, cancel := context.WithTimeout(ctx, origin.Spec.Timeout.Duration)
defer cancel()
Expand Down Expand Up @@ -352,7 +339,6 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr

debuglog.Info("ran updates to working dir", "working", tmp)

var statusMessage string
var signingEntity *openpgp.Entity
if gitSpec.Commit.SigningKey != nil {
if signingEntity, err = r.getSigningEntity(ctx, auto); err != nil {
Expand Down Expand Up @@ -386,6 +372,7 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
err = extgogit.ErrEmptyCommit
}

var statusMessage string
if err != nil {
if !errors.Is(err, git.ErrNoStagedFiles) && !errors.Is(err, extgogit.ErrEmptyCommit) {
return failWithError(err)
Expand All @@ -401,20 +388,39 @@ func (r *ImageUpdateAutomationReconciler) Reconcile(ctx context.Context, req ctr
// Use the git operations timeout for the repo.
pushCtx, cancel := context.WithTimeout(ctx, origin.Spec.Timeout.Duration)
defer cancel()
opts := repository.PushConfig{}
var pushConfig repository.PushConfig

// Use push refspec if provided.
if gitSpec.Push != nil && gitSpec.Push.Refspec != "" {
pushConfig.Refspecs = []string{gitSpec.Push.Refspec}
}

// If the force push feature flag is true and we are pushing to a
// different branch than the one we checked out to, then force push
// these changes.
forcePush := r.features[features.GitForcePushBranch]
if forcePush && pushBranch != ref.Branch {
opts.Force = true
if forcePush && switchBranch {
pushConfig.Force = true
}
if err := gitClient.Push(pushCtx, opts); err != nil {

if err := gitClient.Push(pushCtx, pushConfig); err != nil {
return failWithError(err)
}

r.event(ctx, auto, eventv1.EventSeverityInfo, fmt.Sprintf("Committed and pushed change %s to %s\n%s", rev, pushBranch, message))
log.Info("pushed commit to origin", "revision", rev, "branch", pushBranch)
if len(pushConfig.Refspecs) > 0 {
r.event(ctx, auto, eventv1.EventSeverityInfo,
fmt.Sprintf("Committed and pushed change %s using refspec %s\n%s", rev, gitSpec.Push.Refspec, message))
log.Info("pushed commit to origin", "revision", rev, "refspec", gitSpec.Push.Refspec)
statusMessage = "committed and pushed " + rev + " using refspec " + gitSpec.Push.Refspec
} else {
r.event(ctx, auto, eventv1.EventSeverityInfo,
fmt.Sprintf("Committed and pushed change %s to %s\n%s", rev, pushBranch, message))
log.Info("pushed commit to origin", "revision", rev, "branch", pushBranch)
statusMessage = "committed and pushed " + rev + " to " + pushBranch
}

auto.Status.LastPushCommit = rev
auto.Status.LastPushTime = &metav1.Time{Time: start}
statusMessage = "committed and pushed " + rev + " to " + pushBranch
}

// Getting to here is a successful run.
Expand Down Expand Up @@ -545,6 +551,38 @@ func (r *ImageUpdateAutomationReconciler) getAuthOpts(ctx context.Context, repos
return opts, nil
}

// constructGitClient constructs and returns a new gogit client.
func (r *ImageUpdateAutomationReconciler) constructGitClient(ctx context.Context,
origin *sourcev1.GitRepository, repoDir string, switchBranch bool) (*gogit.Client, error) {
authOpts, err := r.getAuthOpts(ctx, origin)
if err != nil {
return nil, err
}

clientOpts := []gogit.ClientOption{gogit.WithDiskStorage()}
if authOpts.Transport == git.HTTP {
clientOpts = append(clientOpts, gogit.WithInsecureCredentialsOverHTTP())
}

// If the push branch is different from the checkout ref, we need to
// have all the references downloaded at clone time, to ensure that
// SwitchBranch will have access to the target branch state. fluxcd/flux2#3384
//
// To always overwrite the push branch, the feature gate
// GitAllBranchReferences can be set to false, which will cause
// the SwitchBranch operation to ignore the remote branch state.
allReferences := r.features[features.GitAllBranchReferences]
if switchBranch {
clientOpts = append(clientOpts, gogit.WithSingleBranch(!allReferences))
}

gitClient, err := gogit.NewClient(repoDir, authOpts, clientOpts...)
if err != nil {
return nil, err
}
return gitClient, nil
}

// getSigningEntity retrieves an OpenPGP entity referenced by the
// provided imagev1.ImageUpdateAutomation for git commit signing
func (r *ImageUpdateAutomationReconciler) getSigningEntity(ctx context.Context, auto imagev1.ImageUpdateAutomation) (*openpgp.Entity, error) {
Expand Down
Loading

0 comments on commit 312426b

Please sign in to comment.