Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up checkout by fetching minimal data from remote and lazily loading it later #612

Closed
Tyriar opened this issue Oct 24, 2018 · 7 comments
Labels
feature-request Request for new features or functionality performance

Comments

@Tyriar
Copy link
Member

Tyriar commented Oct 24, 2018

One of my big annoyances with the extension is how long it takes to checkout a branch, particularly on the first time checking this out. It's also even worse on Windows as git seems to be a little slower (due to FS?). This problem is also much worth with larger repos due to the size of the history, branch/tag count, etc.

I'm proposing that when a PR is checked out for the first time we fetch just the branch (r branch) and no history (--depth=1), something like this:

git remote add r
git fetch --depth 1 r branch
git checkout remote_branch

The history and rest of branches can then be lazily fetched later with this:

git fetch --unshallow r

Based on my testing this should provide a very significant speed up:

Normal fetch and checkout:

powershell -Command "Measure-Command {git remote add r https://github.com/Tyriar/xterm.js; git fetch r}"
...
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 2
Milliseconds      : 935
Ticks             : 29353111
TotalDays         : 3.39735081018518E-05
TotalHours        : 0.000815364194444444
TotalMinutes      : 0.0489218516666667
TotalSeconds      : 2.9353111
TotalMilliseconds : 2935.3111

Fetch remote branch only with no history:

powershell -Command "Measure-Command {git remote add r https://github.com/Tyriar/xterm.js; git fetch --depth 1 r 1747_npe}"
...
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 149
Ticks             : 11490152
TotalDays         : 1.3298787037037E-05
TotalHours        : 0.000319170888888889
TotalMinutes      : 0.0191502533333333
TotalSeconds      : 1.1490152
TotalMilliseconds : 1149.0152
@RMacfarlane RMacfarlane added the feature-request Request for new features or functionality label Oct 24, 2018
@Tyriar
Copy link
Member Author

Tyriar commented Oct 24, 2018

The change is even more significant for larger repos, 1733ms vs 6485ms for checkout out a branch on Microsoft/vscode:

Normal fetch and checkout:
C:\Users\daimms.REDMOND\Documents\playground\vscode>powershell -Command "Measure-Command {git remote add r https://github.com/Microsoft/vscode; git fetch r; git checkout -b test r/tyriar/electron-3.0.x-conpty}"
From https://github.com/Microsoft/vscode
 * [new branch]            1.11                       -> r/1.11
 * [new branch]            aeschli/bundled-app-insights -> r/aeschli/bundled-app-insights
 * [new branch]            aeschli/json-tests         -> r/aeschli/json-tests
 * [new branch]            aeschli/tokenizersupport-initialization -> r/aeschli/tokenizersupport-initialization
 * [new branch]            alex/binary-rpc            -> r/alex/binary-rpc
 * [new branch]            alex/snapshot2             -> r/alex/snapshot2
 * [new branch]            alexr00/TaskExitNotFiring  -> r/alexr00/TaskExitNotFiring
 * [new branch]            alexr00/addNpmScriptTooltip -> r/alexr00/addNpmScriptTooltip
 * [new branch]            alexr00/fixWorkspaceRootSplit -> r/alexr00/fixWorkspaceRootSplit
 * [new branch]            alexr00/rerunTask          -> r/alexr00/rerunTask
 * [new branch]            alexr00/revealSilent       -> r/alexr00/revealSilent
 * [new branch]            avoid-fssync               -> r/avoid-fssync
 * [new branch]            build-using-ts-3.0         -> r/build-using-ts-3.0
 * [new branch]            chines-locale              -> r/chines-locale
 * [new branch]            chrmarti/51986             -> r/chrmarti/51986
 * [new branch]            chrmarti/57850             -> r/chrmarti/57850
 * [new branch]            chrmarti/59343             -> r/chrmarti/59343
 * [new branch]            chrmarti/60243             -> r/chrmarti/60243
 * [new branch]            chrmarti/cloudshellts      -> r/chrmarti/cloudshellts
 * [new branch]            chrmarti/embeddedEditorBackground -> r/chrmarti/embeddedEditorBackground
 * [new branch]            chrmarti/quickinputapi     -> r/chrmarti/quickinputapi
 * [new branch]            chrmarti/quickinputapioptions -> r/chrmarti/quickinputapioptions
 * [new branch]            chrmarti/quickinputapiwip  -> r/chrmarti/quickinputapiwip
 * [new branch]            core-lang                  -> r/core-lang
 * [new branch]            css-json-shrinkwrap        -> r/css-json-shrinkwrap
 * [new branch]            danyeh/languageIDfix       -> r/danyeh/languageIDfix
 * [new branch]            dbaeumer/47386             -> r/dbaeumer/47386
 * [new branch]            dbaeumer/48032             -> r/dbaeumer/48032
 * [new branch]            dbaeumer/50588             -> r/dbaeumer/50588
 * [new branch]            dbaeumer/50799             -> r/dbaeumer/50799
 * [new branch]            dbaeumer/54111             -> r/dbaeumer/54111
 * [new branch]            dbaeumer/57148             -> r/dbaeumer/57148
 * [new branch]            dockfind                   -> r/dockfind
 * [new branch]            electron-3.0.x             -> r/electron-3.0.x
 * [new branch]            err-telemetry              -> r/err-telemetry
 * [new branch]            error                      -> r/error
 * [new branch]            file-ext-tag-telemetry     -> r/file-ext-tag-telemetry
 * [new branch]            fix-indent                 -> r/fix-indent
 * [new branch]            for-christof               -> r/for-christof
 * [new branch]            hackathon/profile          -> r/hackathon/profile
 * [new branch]            implementation-progress    -> r/implementation-progress
 * [new branch]            isidorn/buildTest          -> r/isidorn/buildTest
 * [new branch]            isidorn/debugLogging       -> r/isidorn/debugLogging
 * [new branch]            isidorn/lazyFormatter      -> r/isidorn/lazyFormatter
 * [new branch]            isidorn/loadedScriptSchedulers -> r/isidorn/loadedScriptSchedulers
 * [new branch]            isidorn/normalizePaths     -> r/isidorn/normalizePaths
 * [new branch]            isidorn/resourceTrimTrailingSlash -> r/isidorn/resourceTrimTrailingSlash
 * [new branch]            isidorn/taskActiveEvent    -> r/isidorn/taskActiveEvent
 * [new branch]            joao/build                 -> r/joao/build
 * [new branch]            joao/clean-deps            -> r/joao/clean-deps
 * [new branch]            joao/compile-es6           -> r/joao/compile-es6
 * [new branch]            joao/darwin-executable-name -> r/joao/darwin-executable-name
 * [new branch]            joao/data-tree-model       -> r/joao/data-tree-model
 * [new branch]            joao/git-api               -> r/joao/git-api
 * [new branch]            joao/inno-signtool         -> r/joao/inno-signtool
 * [new branch]            joao/ipc-cancellation      -> r/joao/ipc-cancellation
 * [new branch]            joao/ipc-rpc               -> r/joao/ipc-rpc
 * [new branch]            joao/linux-integration-tests -> r/joao/linux-integration-tests
 * [new branch]            joao/log-path              -> r/joao/log-path
 * [new branch]            joao/next                  -> r/joao/next
 * [new branch]            joao/tree                  -> r/joao/tree
 * [new branch]            joao/tree-filter           -> r/joao/tree-filter
 * [new branch]            joao/uninstaller           -> r/joao/uninstaller
 * [new branch]            joao/wip-tree              -> r/joao/wip-tree
 * [new branch]            joaomoreno-patch-1         -> r/joaomoreno-patch-1
 * [new branch]            joh/fix59538               -> r/joh/fix59538
 * [new branch]            joh/log-appender           -> r/joh/log-appender
 * [new branch]            joh/next                   -> r/joh/next
 * [new branch]            joh/perf-canary            -> r/joh/perf-canary
 * [new branch]            joh/search-cancel          -> r/joh/search-cancel
 * [new branch]            joh/webpack-json           -> r/joh/webpack-json
 * [new branch]            kieferrm/errortel          -> r/kieferrm/errortel
 * [new branch]            kieferrm/mainDebug         -> r/kieferrm/mainDebug
 * [new branch]            loc/1.24.0                 -> r/loc/1.24.0
 * [new branch]            master                     -> r/master
 * [new branch]            misolori/a11y-themes       -> r/misolori/a11y-themes
 * [new branch]            misolori/activity-bar-update -> r/misolori/activity-bar-update
 * [new branch]            misolori/light-theme-icons -> r/misolori/light-theme-icons
 * [new branch]            misolori/light-theme-icons-selection-dont-work -> r/misolori/light-theme-icons-selection-dont-work
 * [new branch]            misolori/welcome-screen-keys -> r/misolori/welcome-screen-keys
 * [new branch]            mjbvz/remove-std-fork      -> r/mjbvz/remove-std-fork
 * [new branch]            mjbvz/trigger-context      -> r/mjbvz/trigger-context
 * [new branch]            octref/58555               -> r/octref/58555
 * [new branch]            optout-locale              -> r/optout-locale
 * [new branch]            php-html-enter-rules       -> r/php-html-enter-rules
 * [new branch]            prompt-lang-pack           -> r/prompt-lang-pack
 * [new branch]            ramyar/copy-suggest-docs   -> r/ramyar/copy-suggest-docs
 * [new branch]            ramyar/reload              -> r/ramyar/reload
 * [new branch]            ramyar/suggest-no-icons    -> r/ramyar/suggest-no-icons
 * [new branch]            rebornix/ESM               -> r/rebornix/ESM
 * [new branch]            rebornix/FastTokenization  -> r/rebornix/FastTokenization
 * [new branch]            rebornix/nativeWordOperation -> r/rebornix/nativeWordOperation
 * [new branch]            rebornix/review            -> r/rebornix/review
 * [new branch]            release/1.14               -> r/release/1.14
 * [new branch]            release/1.15               -> r/release/1.15
 * [new branch]            release/1.16               -> r/release/1.16
 * [new branch]            release/1.17               -> r/release/1.17
 * [new branch]            release/1.18               -> r/release/1.18
 * [new branch]            release/1.19               -> r/release/1.19
 * [new branch]            release/1.20               -> r/release/1.20
 * [new branch]            release/1.21               -> r/release/1.21
 * [new branch]            release/1.22               -> r/release/1.22
 * [new branch]            release/1.23               -> r/release/1.23
 * [new branch]            release/1.24               -> r/release/1.24
 * [new branch]            release/1.25               -> r/release/1.25
 * [new branch]            release/1.26               -> r/release/1.26
 * [new branch]            release/1.27               -> r/release/1.27
 * [new branch]            release/1.28               -> r/release/1.28
 * [new branch]            reuse-focuseditem          -> r/reuse-focuseditem
 * [new branch]            revert-52782-electron-2.0.x -> r/revert-52782-electron-2.0.x
 * [new branch]            revert-54854-revert-52782-electron-2.0.x -> r/revert-54854-revert-52782-electron-2.0.x
 * [new branch]            rmacfarlane/delete-comments -> r/rmacfarlane/delete-comments
 * [new branch]            rmacfarlane/unsupported    -> r/rmacfarlane/unsupported
 * [new branch]            roblou/34711               -> r/roblou/34711
 * [new branch]            roblou/49039               -> r/roblou/49039
 * [new branch]            roblou/55649               -> r/roblou/55649
 * [new branch]            roblou/cachedSearchProvider -> r/roblou/cachedSearchProvider
 * [new branch]            roblou/caseInsensitiveGlobs -> r/roblou/caseInsensitiveGlobs
 * [new branch]            roblou/ehSearch            -> r/roblou/ehSearch
 * [new branch]            roblou/fileSearchChanges   -> r/roblou/fileSearchChanges
 * [new branch]            roblou/fileTextSearchSep   -> r/roblou/fileTextSearchSep
 * [new branch]            roblou/findFilesUri        -> r/roblou/findFilesUri
 * [new branch]            roblou/promoteExtLogAPI    -> r/roblou/promoteExtLogAPI
 * [new branch]            roblou/searchAPIchanges    -> r/roblou/searchAPIchanges
 * [new branch]            roblou/settingsEditorAsTree -> r/roblou/settingsEditorAsTree
 * [new branch]            roblou/settingsSearchIdChanges -> r/roblou/settingsSearchIdChanges
 * [new branch]            sandy081/54315             -> r/sandy081/54315
 * [new branch]            sandy081/54746             -> r/sandy081/54746
 * [new branch]            sandy081/55879             -> r/sandy081/55879
 * [new branch]            sandy081/59603             -> r/sandy081/59603
 * [new branch]            sandy081/openUri           -> r/sandy081/openUri
 * [new branch]            sandy081/treeItemDecorations -> r/sandy081/treeItemDecorations
 * [new branch]            settings-contrib           -> r/settings-contrib
 * [new branch]            settings-polish            -> r/settings-polish
 * [new branch]            show-english               -> r/show-english
 * [new branch]            standalone/0.1.1           -> r/standalone/0.1.1
 * [new branch]            standalone/0.1.x           -> r/standalone/0.1.x
 * [new branch]            standalone/0.10.x          -> r/standalone/0.10.x
 * [new branch]            standalone/0.11.x          -> r/standalone/0.11.x
 * [new branch]            standalone/0.12.x          -> r/standalone/0.12.x
 * [new branch]            standalone/0.13.x          -> r/standalone/0.13.x
 * [new branch]            standalone/0.14.x          -> r/standalone/0.14.x
 * [new branch]            standalone/0.15.x          -> r/standalone/0.15.x
 * [new branch]            standalone/0.6.x           -> r/standalone/0.6.x
 * [new branch]            standalone/0.7.x           -> r/standalone/0.7.x
 * [new branch]            standalone/0.8.x           -> r/standalone/0.8.x
 * [new branch]            triggerExpansionOnTab      -> r/triggerExpansionOnTab
 * [new branch]            tsalinger/multipath-handling-quoted-backslash-in-linux -> r/tsalinger/multipath-handling-quoted-backslash-in-linux
 * [new branch]            tsalinger/multipath-reveal-children -> r/tsalinger/multipath-reveal-children
 * [new branch]            tyriar/54093_buffering     -> r/tyriar/54093_buffering
 * [new branch]            tyriar/dom_render          -> r/tyriar/dom_render
 * [new branch]            tyriar/electron-1.8.2      -> r/tyriar/electron-1.8.2
 * [new branch]            tyriar/electron-3.0.x-conpty -> r/tyriar/electron-3.0.x-conpty
 * [new branch]            tyriar/snap                -> r/tyriar/snap
 * [new branch]            tyriar/term_ligatures      -> r/tyriar/term_ligatures
 * [new branch]            tyriar/term_ligatures2     -> r/tyriar/term_ligatures2
 * [new branch]            tyriar/term_tele           -> r/tyriar/term_tele
 * [new branch]            umd                        -> r/umd
 * [new branch]            update-event               -> r/update-event
Switched to a new branch 'test'


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 6
Milliseconds      : 485
Ticks             : 64858838
TotalDays         : 7.5068099537037E-05
TotalHours        : 0.00180163438888889
TotalMinutes      : 0.108098063333333
TotalSeconds      : 6.4858838
TotalMilliseconds : 6485.8838

Fetch remote branch only with no history:

C:\Users\daimms.REDMOND\Documents\playground\vscode>powershell -Command "Measure-Command {git remote add r https://github.com/Microsoft/vscode; git fetch --depth 1 r tyriar/electron-3.0.x-conpty}"
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From https://github.com/Microsoft/vscode
 * branch                  tyriar/electron-3.0.x-conpty -> FETCH_HEAD
 * [new branch]            tyriar/electron-3.0.x-conpty -> r/tyriar/electron-3.0.x-conpty


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 733
Ticks             : 17337568
TotalDays         : 2.00666296296296E-05
TotalHours        : 0.000481599111111111
TotalMinutes      : 0.0288959466666667
TotalSeconds      : 1.7337568
TotalMilliseconds : 1733.7568

@rebornix
Copy link
Member

We added some log data for checkout process. This morning I see a huge lag when checkout to a branch, the information is not enough yet, I'm going to add more later but it already shows some good info

[Debug 956677s 804ms] Review> there is already one local branch usernamehw/pr/usernamehw/59193 associated with Pull Request #59193
[Debug 956721s 810ms] Found GitHub remote
[Debug 956723s 749ms] GitHelper> associate pr/usernamehw/59193 with Pull Request #59193
[Debug 956725s 892ms] Found GitHub remote
[Debug 956726s 416ms] Review> current branch pr/usernamehw/59193 is associated with pull request #59193
[Debug 956727s 783ms] Review> register comments provider
[Debug 956727s 784ms] Review> display pull request status bar indicator and refresh pull request tree view.
[Debug 956727s 814ms] Found GitHub remote

44 seconds between the first and second log ;(

@kieferrm kieferrm mentioned this issue Nov 12, 2018
16 tasks
@rebornix
Copy link
Member

Fetching the branch without history is fast and we can probably fetch the history in background later but the catch here is the diff view relies on mergebase to display a correct content diff, to get a correct mergebase we need to fetch the whole history of the target branch and the pr branch. So the blocker here is #698, if that one is not fixed, the "Contents Changes in Pull Request" would be as slow as before.

If I understand correctly, the benefits of this is users get the code as soon as possible and then if they have some watch task running locally, they can the build result earlier.

@Tyriar
Copy link
Member Author

Tyriar commented Nov 26, 2018

Just capturing our offline discussion here: I don't think it's blocked on #698, I just think the switching to review mode should be split into 2 steps, the first of which checks out the minimal set of code to unblock the user to view the code, run it, etc., the second fetches the history to enable diffs, etc.

@kieferrm kieferrm mentioned this issue Dec 21, 2018
20 tasks
@rebornix
Copy link
Member

Fun fact, we can't fetch into current branch ;)

git fetch --depth 1 origin dev
git checkout dev
git fetch --unshallow dev

will throw error fatal: Refusing to fetch into current branch refs/heads/develop of non-bare repository. More details please read https://stackoverflow.com/questions/29028696/why-git-fetch-origin-branchbranch-works-only-on-a-non-current-branch .

However we can do a git pull --unshallow to fetch all the history. I'll send a pr to VSCode Git to support this.

@Tyriar
Copy link
Member Author

Tyriar commented Jan 7, 2019

@rebornix is all done for this on the extension side? Looking forward to testing it out 🙂

@rebornix
Copy link
Member

rebornix commented Jan 7, 2019

@Tyriar some work still needs to be done from extension side, will send out a pr.

@Tyriar Tyriar reopened this Jan 7, 2019
rebornix added a commit that referenced this issue Jan 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request Request for new features or functionality performance
Projects
None yet
Development

No branches or pull requests

3 participants