-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use singleflight to clone/update repository cache #5171
Conversation
Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5171 +/- ##
=======================================
Coverage 22.90% 22.91%
=======================================
Files 416 416
Lines 44598 44604 +6
=======================================
+ Hits 10216 10219 +3
- Misses 33592 33594 +2
- Partials 790 791 +1 ☔ View full report in Codecov by Sentry. |
_, err, _ := c.repoSingleFlights.Do(repoID, func() (interface{}, error) { | ||
_, err := os.Stat(repoCachePath) | ||
if err != nil && !os.IsNotExist(err) { | ||
return nil, err | ||
} | ||
out, err := retryCommand(3, time.Second, logger, func() ([]byte, error) { | ||
args := []string{"clone", "--mirror", remote, repoCachePath} | ||
args = append(authArgs, args...) | ||
return runGitCommand(ctx, c.gitPath, "", c.envsForRepo(remote), args...) | ||
}) | ||
if err != nil { | ||
logger.Error("failed to clone from remote", | ||
zap.String("out", string(out)), | ||
zap.Error(err), | ||
) | ||
return nil, fmt.Errorf("failed to clone from remote: %v", err) | ||
} | ||
} else { | ||
// Cache hit. Do a git fetch to keep updated. | ||
c.logger.Info(fmt.Sprintf("fetching %s to update the cache", repoID)) | ||
out, err := retryCommand(3, time.Second, c.logger, func() ([]byte, error) { | ||
args := []string{"fetch"} | ||
args = append(authArgs, args...) | ||
return runGitCommand(ctx, c.gitPath, repoCachePath, c.envsForRepo(remote), args...) | ||
}) | ||
if err != nil { | ||
logger.Error("failed to fetch from remote", | ||
zap.String("out", string(out)), | ||
zap.Error(err), | ||
) | ||
return nil, fmt.Errorf("failed to fetch: %v", err) | ||
|
||
if os.IsNotExist(err) { | ||
// Cache miss, clone for the first time. | ||
logger.Info(fmt.Sprintf("cloning %s for the first time", repoID)) | ||
if err := os.MkdirAll(filepath.Dir(repoCachePath), os.ModePerm); err != nil && !os.IsExist(err) { | ||
return nil, err | ||
} | ||
out, err := retryCommand(3, time.Second, logger, func() ([]byte, error) { | ||
args := []string{"clone", "--mirror", remote, repoCachePath} | ||
args = append(authArgs, args...) | ||
return runGitCommand(ctx, c.gitPath, "", c.envsForRepo(remote), args...) | ||
}) | ||
if err != nil { | ||
logger.Error("failed to clone from remote", | ||
zap.String("out", string(out)), | ||
zap.Error(err), | ||
) | ||
return nil, fmt.Errorf("failed to clone from remote: %v", err) | ||
} | ||
} else { | ||
// Cache hit. Do a git fetch to keep updated. | ||
c.logger.Info(fmt.Sprintf("fetching %s to update the cache", repoID)) | ||
out, err := retryCommand(3, time.Second, c.logger, func() ([]byte, error) { | ||
args := []string{"fetch"} | ||
args = append(authArgs, args...) | ||
return runGitCommand(ctx, c.gitPath, repoCachePath, c.envsForRepo(remote), args...) | ||
}) | ||
if err != nil { | ||
logger.Error("failed to fetch from remote", | ||
zap.String("out", string(out)), | ||
zap.Error(err), | ||
) | ||
return nil, fmt.Errorf("failed to fetch: %v", err) | ||
} | ||
} | ||
return nil, nil | ||
}) | ||
if err != nil { | ||
return nil, err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[ask] If piped has already cloned the repo, then it hasn't fetched some updates?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, piped do a git fetch
at L186
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If so, we might consider the effect which piped doesn't fetch the repo at this point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the cache doesn't exist, piped does git clone
. Else, does git fetch.
Oops, are you considering the race condition such as the new updates are pushed after git fetch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I expected that the code inside the singleflight.Do
are executed at first call.
The scope of the singlefight seems from L156 to L197. So in my understanding, at first, singleflight.Do
executes the method and clone successfully and recorded the result with the repoID
. And the second time, singleflight.Do
just return the recorded values, not execute the code inside the singleflight.Do
.
Sorry if I misunderstood 🙏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's the behavior of the sync.Once
, not the singleflight.Do
.
singleflight.Do
treats only the duplicate calls that occur when the former call is in-flight.
https://pkg.go.dev/internal/singleflight#Group.Do
Do executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results. The return value shared indicates whether v was given to multiple callers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I see. Thank you. I misunderstood the behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It means that with one execution for sigleflight.Do
, the goroutines executed at the same time can confirm one result.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Also if possible, we should describe the result of the pprof
Thank you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greate improvement 👍
Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]> Signed-off-by: pipecd-bot <[email protected]>
Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]> Signed-off-by: pipecd-bot <[email protected]>
* Modified to use Git with PAT (#4571) * fix to read PAT settings from file Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * piped Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * include PAT information in URL Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: modification of conditional branching Co-authored-by: sivchari <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: corrected error in error message Co-authored-by: sivchari <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: integration of mask function Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: make validation test Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: function name Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: rename function for validation PAT Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: fix test code as pointed out in the review Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * feat: add explan for git personal access token in document Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: change required in documentation Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: change return value Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: add test case Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: fix test Signed-off-by: swallow <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: PipedGit struct to use password authentication instead of personal access token Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix to read PAT settings from file Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * piped Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: integration of mask function Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: make validation test Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: function name Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: rename function for validation PAT Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: fix test Signed-off-by: swallow <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: PipedGit struct to use password authentication instead of personal access token Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Fix Git authentication configuration Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Update password authentication configuration Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Fix error variable name Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Fix rename password Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Refactor includePasswordAuthRemote function Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Update password authentication in clone test Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: delete PasswordAuth Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: remove unused PasswordAuth field and refactor password authentication in git client Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * Remove unnecessary print statement in Validate function Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: fix code for rebase Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * fix: remove unused GitPasswordAuth configuration Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: sZma5a <[email protected]> * feat: add password decoding for password in includePasswordRemote function Signed-off-by: sZma5a <[email protected]> * fix: refactor Git password authentication method Signed-off-by: sZma5a <[email protected]> * fix: update password encoding in TestCloneUsingPassword Signed-off-by: sZma5a <[email protected]> * Update docs/content/en/docs-dev/user-guide/managing-piped/configuration-reference.md Co-authored-by: Yoshiki Fujikane <[email protected]> Signed-off-by: sZma5a <[email protected]> * Update pkg/config/piped.go Co-authored-by: Yoshiki Fujikane <[email protected]> Signed-off-by: sZma5a <[email protected]> * [wip] delete password Signed-off-by: sZma5a <[email protected]> * [wip] not tested - change token to args from url Signed-off-by: sZma5a <[email protected]> * Fix commented out test case Signed-off-by: sZma5a <[email protected]> * Refactor authentication in git client Signed-off-by: sZma5a <[email protected]> * feat: add password decoding function and replace Password string Signed-off-by: sZma5a <[email protected]> --------- Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Co-authored-by: sZma5a <[email protected]> Co-authored-by: sivchari <[email protected]> Co-authored-by: 鈴木 優耀 <[email protected]> Co-authored-by: Your Name <[email protected]> Co-authored-by: Yoshiki Fujikane <[email protected]> Signed-off-by: pipecd-bot <[email protected]> * Use singleflight to clone/update repository cache (#5171) Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]> Signed-off-by: pipecd-bot <[email protected]> * Refactor the git clone (#5190) Move the authArgs into the singleflight closure, as they are only used within it. Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]> Signed-off-by: pipecd-bot <[email protected]> --------- Signed-off-by: sZma5a <[email protected]> Signed-off-by: 鈴木 優耀 <[email protected]> Signed-off-by: Your Name <[email protected]> Signed-off-by: sZma5a <[email protected]> Signed-off-by: swallow <[email protected]> Signed-off-by: pipecd-bot <[email protected]> Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]> Co-authored-by: sZma5a <[email protected]> Co-authored-by: sZma5a <[email protected]> Co-authored-by: sivchari <[email protected]> Co-authored-by: 鈴木 優耀 <[email protected]> Co-authored-by: Your Name <[email protected]> Co-authored-by: Yoshiki Fujikane <[email protected]> Co-authored-by: Shinnosuke Sawada-Dazai <[email protected]>
What this PR does / why we need it:
I profiled goroutines with the debug/pprof/goroutine endpoint and noticed goroutines are waiting at the lockRepo method.
goroutines waiting at the lockRepo
I triggered 50 deployments simultaneously, and 45 goroutines are waiting at the lockRepo method.The lock/unlock line is here.
pipecd/pkg/git/client.go
Lines 153 to 154 in a61c397
The git clone/fetch operation is heavy, so we can put these in the singleflight, not guarding with the mutex.
I got the performance improvement on my local machine, about 10x 〜 20x faster when triggering 50 deployments simultaneously.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: