fix: git compatibility issue with large repository on windows #947

scolladon · 2024-11-07T21:39:54Z

Explain your changes

Implements Git Adapter using only simpleGit
Remove implementation using isomorphic-git and simplify treatment

Implementation is now a little bit simpler
But it is also slower and more dependant of git behavior in the terminal

Does this close any currently open issues?

closes #862

Jest tests added to cover the fix.

codecov · 2024-11-07T21:41:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (d241475) to head (1d5e6f4).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #947   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           40        40           
  Lines         1083      1029   -54     
  Branches       115       103   -12     
=========================================
- Hits          1083      1029   -54

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

scolladon · 2024-11-07T21:47:28Z

Hi @kez-sophos @ALesc21 @dsweetmanusda !

Could you test on your side and help us validate this PR please ?
You can follow those steps to easily do it on your side

src/adapter/GitAdapter.ts

scolladon · 2024-11-09T17:17:32Z

Performance test

Current Pull request (`simplegit` git adapter)

e2e-test x 0.80 ops/sec ±7.74% (6 runs sampled)
e2e-test x 0.80 ops/sec ±6.30% (6 runs sampled)
e2e-test x 0.76 ops/sec ±4.39% (6 runs sampled)
e2e-test x 0.77 ops/sec ±4.35% (6 runs sampled)
e2e-test x 0.79 ops/sec ±2.67% (6 runs sampled)
standard deviation 1.62481

Current main `v5.49.2` (`isomorphic-git` git adapter)

e2e-test x 0.73 ops/sec ±16.35% (6 runs sampled)
e2e-test x 0.79 ops/sec ±13.19% (7 runs sampled)
e2e-test x 0.88 ops/sec ±2.68% (7 runs sampled)
e2e-test x 0.90 ops/sec ±1.62% (7 runs sampled)
e2e-test x 0.87 ops/sec ±3.07% (7 runs sampled)
standard deviation 6.40625

Synthesis

Fastest comparison: simplegit is slower by 11.1%
Slowest comparison: simplegit is faster by 4%
Mean comparison: simplegit is slower by 8%
simplegit implementation is more stable in term of duration of execution

On very large repository I noticed ~30% improvements in favor of simplegit implementation

ALesc21 · 2024-11-11T06:22:16Z

HI @scolladon, should I install the new version with the command below and provide the output?

sfdx plugins:install sfdx-git-delta@dev-947

scolladon · 2024-11-11T08:25:28Z

HI @scolladon, should I install the new version with the command below and provide the output?

sfdx plugins:install sfdx-git-delta@dev-947

Yes @ALesc21 !
This way you'll be able to test the version of this PR.
And then you can come back to the previous one by executing:

sfdx plugins:install sfdx-git-delta@latest

codeclimate · 2024-11-11T10:20:51Z

src/adapter/GitAdapter.ts

+      .map(line => treatPathSep(line))
+  }
+
+  public async getFilesPath(path: string): Promise<string[]> {


Function getFilesPath has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

ALesc21 · 2024-11-11T12:55:14Z

hi @scolladon,

after the installation of the new version I run the delta command and could build successfully a package with the latest api version.
So for us the fix is working on the server now. Great job ;)

$ sf sgd:source:delta --to HEAD --from HEAD^ --output deployment/ONECRM-0002
 »   Warning: @salesforce/cli update available from 2.62.6 to 2.64.8.
(node:12296) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instea
d.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:12296) Warning: Deprecated config name: apiVersion. Please use org-api-version instead.
{
  "error": null,
  "output": "deployment/ONECRM-0002",
  "success": true,
  "warnings": [
    "[INFORMATION] sfdx-git-delta v6 is coming soon! Read more and plan your migration: https://github.com/scolladon/sfd
x-git-delta/issues/936"
  ]
}

codeclimate · 2024-11-11T13:11:36Z

Code Climate has analyzed commit 1d5e6f4 and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1

View more on Code Climate.

scolladon · 2024-11-11T13:38:41Z

Thank you very much for your contribution @ALesc21
I just wrote an optimised version, could you repeat the test from the start and tell me the result on your side please ? (reinstall the save dev package and rerun your command please)

Feel free to contribute as well @kez-sophos & @dsweetmanusda ! Your feedback will be really valuable and I hope it will fix your situation as well.

ALesc21 · 2024-11-11T14:55:26Z

hi @scolladon, I reinstalled the version and tested it again. It is still working with your fix for me on the windows server. Below you can find the steps (install dev version, check plugin, generate the package)

DEBIE1TC_SFDX@MSE-SVCRM01 MINGW64 /c/BuildAgent01/work/86775e496ec9c505 (feature/ONECRM-8856|SPARSE)
$ sfdx plugins:install sfdx-git-delta@dev-947
 »   Warning: @salesforce/cli update available from 2.62.6 to 2.64.8.
(node:11880) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instea
d.
(Use `node --trace-deprecation ...` to show where the warning was created)
? sfdx-git-delta isn't signed by Salesforce. Only install the plugin if you trust its creator. Do you want to continue
the installation? yes
Because you approved this plugin, you can avoid future installation confirmations by adding the plugin to the unsignedPl
uginAllowList.json file. For details, see https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/s
fdx_setup_allowlist.htm.
Finished digital signature check.
removed 11 packages, and changed 1 package in 2s
@salesforce/cli: Installing plugin sfdx-git-delta@dev-947... installed v5.49.2

DEBIE1TC_SFDX@MSE-SVCRM01 MINGW64 /c/BuildAgent01/work/86775e496ec9c505 (feature/ONECRM-8856|SPARSE)
$ sf plugins --core
 »   Warning: @salesforce/cli update available from 2.62.6 to 2.64.8.
(node:6060) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead
.
(Use `node --trace-deprecation ...` to show where the warning was created)
@oclif/plugin-autocomplete 3.2.5 (core)
@oclif/plugin-commands 4.1.3 (core)
@oclif/plugin-help 6.2.14 (core)
@oclif/plugin-not-found 3.2.22 (core)
@oclif/plugin-plugins 5.4.14 (core)
@oclif/plugin-search 1.2.11 (core)
@oclif/plugin-update 4.6.3 (core)
@oclif/plugin-version 2.2.14 (core)
@oclif/plugin-warn-if-update-available 3.1.18 (core)
@oclif/plugin-which 3.2.15 (core)
apex 3.5.1 (core)
api 1.3.1 (core)
auth 3.6.65 (core)
community 3.0.14 (3.0.14)
data 3.7.0 (core)
deploy-retrieve 3.12.17 (core)
info 3.4.9 (core)
limits 3.3.32 (core)
marketplace 1.2.26 (core)
org 4.6.0 (core)
packaging 2.8.11 (core)
schema 3.3.34 (core)
settings 2.3.23 (core)
sfdx-git-delta 5.49.2-dev-947.11779231727-1 (dev-947)
sobject 1.4.41 (core)
source 3.5.21 (core)
telemetry 3.6.15 (core)
templates 56.3.22 (core)
trust 3.7.33 (core)
user 3.5.32 (core)

Uninstalled JIT Plugins:
custom-metadata 3.3.33
dev 2.4.2
devops-center 1.2.24
env 3.0.33
functions 1.23.0
signups 2.5.21
@salesforce/sfdx-plugin-lwc-test 1.2.1
@salesforce/sfdx-scanner 4.6.0

DEBIE1TC_SFDX@MSE-SVCRM01 MINGW64 /c/BuildAgent01/work/86775e496ec9c505 (feature/ONECRM-8856|SPARSE)
$ sf sgd:source:delta --to HEAD --from HEAD^ --output deployment/ONECRM-0002
 »   Warning: @salesforce/cli update available from 2.62.6 to 2.64.8.
(node:6920) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead
.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6920) Warning: Deprecated config name: apiVersion. Please use org-api-version instead.
{
  "error": null,
  "output": "deployment/ONECRM-0002",
  "success": true,
  "warnings": [
    "[INFORMATION] sfdx-git-delta v6 is coming soon! Read more and plan your migration: https://github.com/scolladon/sfd
x-git-delta/issues/936"
  ]
}

github-actions · 2024-11-12T11:23:08Z

Shipped in release v5.49.3.
You can install the new version using the version number or the latest-rc channel

$ sfdx plugins:install sfdx-git-delta@latest-rc
$ sfdx plugins:install [email protected]

Happy incremental deployment!

ALesc21 · 2024-11-13T06:24:47Z

the version 5.49.3 is already installed and running smoothly on the server.
I really appreciate your help and hard work ;)

scolladon added 4 commits November 7, 2024 22:07

refactor: swap git adapter implementation to use simplegit

62d4f25

build: remove isomorphic git dependency

56002de

test: adapt test to the new dependency

ccc7061

fix: binary file handling

cf6261d

scolladon requested a review from mehdicherf as a code owner November 7, 2024 21:39

fix: windows EOL and path sep

8354c77

scolladon force-pushed the fix/windows-git-portability branch 2 times, most recently from c87f656 to 49cecaf Compare November 7, 2024 22:31

refactor: remove non necessary treatPathSep

c637851

scolladon force-pushed the fix/windows-git-portability branch from 49cecaf to c637851 Compare November 7, 2024 22:37

fix: path parameter issue

415c1bb

scolladon force-pushed the fix/windows-git-portability branch from 046731b to 415c1bb Compare November 7, 2024 22:57

scolladon added 7 commits November 8, 2024 08:42

refactor: improve variable names

74d8611

fix: force treating path with posix

7f781d2

refactor: move constants in their file

d65e478

fix: not necessary path handling

e98027b

fix: path resolution depending the OS

7437966

fix: unused variable

120851c

refactor: simplify already catched issues

f90e7fb

scolladon force-pushed the fix/windows-git-portability branch from 71721b7 to 07d0806 Compare November 8, 2024 09:08

codeclimate bot reviewed Nov 8, 2024

View reviewed changes

src/adapter/GitAdapter.ts Outdated Show resolved Hide resolved

scolladon force-pushed the fix/windows-git-portability branch from aa7a566 to b562c3e Compare November 8, 2024 09:40

fix: too long filename on windows

d37205b

scolladon force-pushed the fix/windows-git-portability branch from b562c3e to d37205b Compare November 8, 2024 09:41

refactor: use async generator

e523e24

scolladon force-pushed the fix/windows-git-portability branch from 4dcf568 to e523e24 Compare November 8, 2024 17:53

codeclimate bot reviewed Nov 11, 2024

View reviewed changes

refactor: add caching layer and improve performance

1d5e6f4

scolladon force-pushed the fix/windows-git-portability branch from 2b9dfef to 1d5e6f4 Compare November 11, 2024 13:11

mehdicherf approved these changes Nov 12, 2024

View reviewed changes

scolladon merged commit 4728132 into main Nov 12, 2024
20 of 22 checks passed

scolladon deleted the fix/windows-git-portability branch November 12, 2024 11:05

scolladon mentioned this pull request Nov 12, 2024

chore(main): release 5.49.3 #949

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: git compatibility issue with large repository on windows #947

fix: git compatibility issue with large repository on windows #947

scolladon commented Nov 7, 2024

codecov bot commented Nov 7, 2024 •

edited

Loading

scolladon commented Nov 7, 2024 •

edited

Loading

scolladon commented Nov 9, 2024 •

edited

Loading

ALesc21 commented Nov 11, 2024

scolladon commented Nov 11, 2024

codeclimate bot Nov 11, 2024

ALesc21 commented Nov 11, 2024

codeclimate bot commented Nov 11, 2024

scolladon commented Nov 11, 2024

ALesc21 commented Nov 11, 2024

github-actions bot commented Nov 12, 2024

ALesc21 commented Nov 13, 2024

fix: git compatibility issue with large repository on windows #947

fix: git compatibility issue with large repository on windows #947

Conversation

scolladon commented Nov 7, 2024

Explain your changes

Does this close any currently open issues?

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

scolladon commented Nov 7, 2024 • edited Loading

scolladon commented Nov 9, 2024 • edited Loading

Performance test

Current Pull request (simplegit git adapter)

Current main v5.49.2 (isomorphic-git git adapter)

Synthesis

ALesc21 commented Nov 11, 2024

scolladon commented Nov 11, 2024

codeclimate bot Nov 11, 2024

Choose a reason for hiding this comment

ALesc21 commented Nov 11, 2024

codeclimate bot commented Nov 11, 2024

scolladon commented Nov 11, 2024

ALesc21 commented Nov 11, 2024

github-actions bot commented Nov 12, 2024

ALesc21 commented Nov 13, 2024

codecov bot commented Nov 7, 2024 •

edited

Loading

scolladon commented Nov 7, 2024 •

edited

Loading

scolladon commented Nov 9, 2024 •

edited

Loading

Current Pull request (`simplegit` git adapter)

Current main `v5.49.2` (`isomorphic-git` git adapter)