Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monomorphize allocators for tsserver/public API, just like core compiler #58045

Merged
merged 5 commits into from
Apr 5, 2024
Merged

Monomorphize allocators for tsserver/public API, just like core compiler #58045

merged 5 commits into from
Apr 5, 2024

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Apr 2, 2024 •
edited
Loading

In #51682 and #51880, @rbuckton monomorphized our Nodes and Symbols for the compiler's allocators.

This PR naively copies the property order from the compiler over to the classes we define in services, netting similar performance wins for similar memory costs. This should significantly improve performance of tsserver (every editor) and public API users who do anything more than parse (ts-eslint, ts-loader, ts-jest, rush, nx, the list goes on...).

Here are the latest benchmarks when forcing tsc to use these allocators, which is a good proxy for how this will affect downstream users: #58045 (comment)

Note that this is not a performance improvement for tsc itself; that benefit already happened in TS 5.0!

image

The tsserver benchmarks remain, and should lead to significant improvements in editor performance:

image

A new benchmark tests "tsc via public API" explicitly:

image

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Apr 2, 2024
@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 2, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@jakebailey jakebailey mentioned this pull request Apr 2, 2024
Closed
@jakebailey
Copy link
Member Author

After the results come back, I'll try and rebase this PR such that it targets a branch that has the tsc change in place. The benchmark system is smart enough to compare a PR against its target branch so that will provide better results than comparing two negative results to see which one is less negative.

(I am liking what I'm seeing in the logs so far, though.)

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,662k (± 0.01%) 301,824k (± 0.01%) +6,162k (+ 2.08%) 301,774k 301,845k p=0.005 n=6
Parse Time 3.19s (± 0.65%) 3.24s (± 0.80%) +0.04s (+ 1.36%) 3.21s 3.28s p=0.015 n=6
Bind Time 1.00s (± 0.82%) 0.99s (± 0.52%) ~ 0.99s 1.00s p=0.523 n=6
Check Time 9.76s (± 0.26%) 9.81s (± 0.38%) +0.05s (+ 0.55%) 9.78s 9.88s p=0.028 n=6
Emit Time 8.38s (± 0.22%) 8.44s (± 0.30%) +0.06s (+ 0.76%) 8.41s 8.47s p=0.005 n=6
Total Time 22.33s (± 0.20%) 22.49s (± 0.24%) +0.15s (+ 0.67%) 22.41s 22.56s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,378k (± 0.94%) 199,743k (± 0.94%) +6,365k (+ 3.29%) 197,382k 202,031k p=0.005 n=6
Parse Time 1.65s (± 0.92%) 1.69s (± 1.24%) +0.04s (+ 2.74%) 1.67s 1.72s p=0.005 n=6
Bind Time 0.88s (± 0.86%) 0.89s (± 0.58%) ~ 0.88s 0.89s p=0.069 n=6
Check Time 11.25s (± 0.22%) 11.30s (± 0.65%) ~ 11.21s 11.40s p=0.422 n=6
Emit Time 3.16s (± 0.31%) 3.19s (± 5.20%) ~ 3.02s 3.36s p=1.000 n=6
Total Time 16.93s (± 0.22%) 17.06s (± 1.06%) ~ 16.81s 17.26s p=0.297 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,426k (± 0.01%) 353,378k (± 0.01%) +5,952k (+ 1.71%) 353,343k 353,466k p=0.005 n=6
Parse Time 3.70s (± 1.33%) 3.60s (± 1.05%) -0.10s (- 2.66%) 3.56s 3.66s p=0.016 n=6
Bind Time 1.37s (± 0.38%) 1.59s (± 3.06%) 🔻+0.22s (+16.02%) 1.50s 1.64s p=0.004 n=6
Check Time 10.20s (± 0.18%) 10.18s (± 0.38%) ~ 10.13s 10.24s p=0.228 n=6
Emit Time 6.03s (± 0.29%) 6.05s (± 0.59%) ~ 6.02s 6.12s p=0.145 n=6
Total Time 21.30s (± 0.18%) 21.43s (± 0.46%) +0.13s (+ 0.59%) 21.30s 21.60s p=0.020 n=6
TFS - node (v18.15.0, x64)
Memory used 302,830k (± 0.01%) 308,523k (± 0.01%) +5,693k (+ 1.88%) 308,500k 308,553k p=0.005 n=6
Parse Time 2.42s (± 0.61%) 2.44s (± 1.03%) ~ 2.42s 2.48s p=0.075 n=6
Bind Time 1.20s (± 0.43%) 1.16s (±10.27%) ~ 1.00s 1.25s p=0.366 n=6
Check Time 7.55s (± 0.61%) 7.62s (± 1.52%) ~ 7.52s 7.78s p=0.469 n=6
Emit Time 4.27s (± 0.52%) 4.31s (± 0.64%) ~ 4.28s 4.35s p=0.052 n=6
Total Time 15.44s (± 0.31%) 15.54s (± 0.22%) +0.10s (+ 0.62%) 15.50s 15.59s p=0.016 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,164k (± 0.01%) 517,146k (± 0.00%) +6,982k (+ 1.37%) 517,117k 517,184k p=0.005 n=6
Parse Time 3.93s (± 0.35%) 3.98s (± 1.13%) +0.06s (+ 1.40%) 3.95s 4.06s p=0.009 n=6
Bind Time 1.45s (± 0.83%) 1.50s (± 1.31%) +0.05s (+ 3.44%) 1.47s 1.52s p=0.006 n=6
Check Time 25.41s (± 0.42%) 25.36s (± 0.42%) ~ 25.23s 25.51s p=0.470 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.79s (± 0.35%) 30.84s (± 0.40%) ~ 30.67s 31.03s p=0.575 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,999k (± 0.00%) 1,750,707k (± 0.00%) +5,708k (+ 0.33%) 1,750,659k 1,750,781k p=0.005 n=6
Parse Time 7.80s (± 0.21%) 8.02s (± 2.40%) +0.22s (+ 2.84%) 7.91s 8.41s p=0.004 n=6
Bind Time 2.82s (± 0.30%) 2.29s (± 1.42%) 🟩-0.53s (-18.88%) 2.27s 2.35s p=0.004 n=6
Check Time 67.07s (± 0.44%) 68.89s (± 1.49%) +1.82s (+ 2.72%) 67.54s 69.74s p=0.005 n=6
Emit Time 0.16s (± 2.52%) 0.16s (± 2.58%) ~ 0.15s 0.16s p=0.218 n=6
Total Time 77.86s (± 0.38%) 79.36s (± 1.11%) +1.50s (+ 1.93%) 78.05s 80.15s p=0.008 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,399,823k (± 0.02%) 2,489,901k (± 0.03%) +90,078k (+ 3.75%) 2,489,254k 2,490,841k p=0.005 n=6
Parse Time 6.02s (± 0.70%) 6.29s (± 0.90%) 🔻+0.27s (+ 4.43%) 6.20s 6.35s p=0.005 n=6
Bind Time 2.26s (± 1.10%) 2.04s (± 1.50%) 🟩-0.22s (- 9.72%) 1.99s 2.07s p=0.005 n=6
Check Time 39.95s (± 0.62%) 40.49s (± 0.15%) +0.55s (+ 1.37%) 40.43s 40.60s p=0.005 n=6
Emit Time 3.17s (± 1.04%) 3.18s (± 1.41%) ~ 3.12s 3.23s p=0.810 n=6
Total Time 51.43s (± 0.39%) 52.01s (± 0.23%) +0.58s (+ 1.12%) 51.85s 52.12s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,131k (± 0.01%) 421,766k (± 0.02%) +5,635k (+ 1.35%) 421,693k 421,940k p=0.005 n=6
Parse Time 2.76s (± 0.37%) 2.74s (± 2.70%) ~ 2.65s 2.82s p=0.418 n=6
Bind Time 1.08s (± 0.38%) 1.16s (± 4.12%) 🔻+0.08s (+ 7.73%) 1.09s 1.22s p=0.004 n=6
Check Time 15.42s (± 0.37%) 15.47s (± 0.34%) ~ 15.41s 15.56s p=0.109 n=6
Emit Time 1.12s (± 1.35%) 1.12s (± 2.24%) ~ 1.09s 1.15s p=0.935 n=6
Total Time 20.38s (± 0.33%) 20.49s (± 0.28%) +0.12s (+ 0.58%) 20.43s 20.60s p=0.019 n=6
vscode - node (v18.15.0, x64)
Memory used 2,899,596k (± 0.00%) 2,901,558k (± 0.00%) +1,962k (+ 0.07%) 2,901,446k 2,901,653k p=0.005 n=6
Parse Time 10.84s (± 0.43%) 11.08s (± 0.51%) +0.24s (+ 2.20%) 11.03s 11.17s p=0.005 n=6
Bind Time 3.48s (± 0.82%) 3.46s (± 0.35%) ~ 3.45s 3.48s p=0.411 n=6
Check Time 61.97s (± 0.74%) 62.10s (± 0.34%) ~ 61.82s 62.46s p=0.173 n=6
Emit Time 16.46s (± 0.52%) 16.58s (± 0.21%) ~ 16.54s 16.63s p=0.054 n=6
Total Time 92.75s (± 0.51%) 93.23s (± 0.22%) ~ 92.90s 93.51s p=0.066 n=6
webpack - node (v18.15.0, x64)
Memory used 408,862k (± 0.01%) 415,521k (± 0.01%) +6,659k (+ 1.63%) 415,441k 415,603k p=0.005 n=6
Parse Time 3.24s (± 0.30%) 3.33s (± 0.75%) +0.10s (+ 2.99%) 3.31s 3.38s p=0.004 n=6
Bind Time 1.41s (± 0.73%) 1.38s (± 0.59%) -0.02s (- 1.66%) 1.37s 1.39s p=0.008 n=6
Check Time 14.31s (± 0.34%) 14.44s (± 0.13%) +0.13s (+ 0.87%) 14.42s 14.46s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.96s (± 0.20%) 19.16s (± 0.11%) +0.20s (+ 1.05%) 19.12s 19.18s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 513,453k (± 0.02%) 531,453k (± 0.01%) +18,000k (+ 3.51%) 531,373k 531,524k p=0.005 n=6
Parse Time 3.93s (± 0.47%) 3.98s (± 0.37%) +0.05s (+ 1.14%) 3.95s 3.99s p=0.012 n=6
Bind Time 1.88s (± 0.65%) 1.67s (± 1.22%) 🟩-0.21s (-11.02%) 1.64s 1.69s p=0.005 n=6
Check Time 3.42s (± 1.02%) 3.60s (± 0.82%) 🔻+0.19s (+ 5.46%) 3.55s 3.64s p=0.005 n=6
Emit Time 0.09s (± 5.95%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.112 n=6
Total Time 9.32s (± 0.45%) 9.33s (± 0.60%) ~ 9.23s 9.39s p=0.470 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,562ms (± 0.27%) 3,524ms (± 0.44%) -38ms (- 1.07%) 3,504ms 3,547ms p=0.005 n=6
Req 2 - geterr 8,322ms (± 0.64%) 7,603ms (± 0.46%) 🟩-719ms (- 8.64%) 7,553ms 7,647ms p=0.005 n=6
Req 3 - references 475ms (± 0.34%) 470ms (± 1.05%) -5ms (- 1.09%) 463ms 475ms p=0.036 n=6
Req 4 - navto 407ms (± 0.18%) 410ms (± 2.59%) ~ 398ms 427ms p=0.373 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 129ms (± 1.44%) 112ms (± 1.31%) 🟩-17ms (-13.27%) 111ms 114ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,223ms (±11.44%) 3,301ms (±11.73%) ~ 2,935ms 3,675ms p=0.471 n=6
Req 2 - geterr 6,135ms (± 1.48%) 5,302ms (±10.57%) 🟩-833ms (-13.58%) 4,570ms 5,729ms p=0.005 n=6
Req 3 - references 472ms (± 6.82%) 441ms (± 9.07%) ~ 379ms 475ms p=0.126 n=6
Req 4 - navto 430ms (± 7.48%) 390ms (±10.69%) 🟩-41ms (- 9.49%) 337ms 434ms p=0.030 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 108ms (±10.76%) 109ms (± 1.00%) ~ 107ms 110ms p=1.000 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,634ms (± 0.20%) 2,596ms (± 0.63%) -39ms (- 1.47%) 2,577ms 2,614ms p=0.005 n=6
Req 2 - geterr 1,739ms (± 2.95%) 1,539ms (± 0.83%) 🟩-200ms (-11.49%) 1,521ms 1,551ms p=0.005 n=6
Req 3 - references 117ms (± 9.69%) 114ms (± 5.65%) ~ 101ms 118ms p=0.297 n=6
Req 4 - navto 369ms (± 1.12%) 365ms (± 1.47%) ~ 357ms 374ms p=0.052 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 307ms (± 2.60%) 291ms (± 0.49%) 🟩-16ms (- 5.06%) 289ms 293ms p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 182.54ms (± 0.18%) 277.41ms (± 0.31%) 🔻+94.87ms (+51.97%) 269.96ms 286.39ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 226.49ms (± 0.17%) 227.25ms (± 0.20%) +0.76ms (+ 0.34%) 225.57ms 233.91ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 221.66ms (± 0.15%) 222.40ms (± 0.16%) +0.74ms (+ 0.33%) 220.79ms 227.46ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 330.05ms (± 0.32%) 331.23ms (± 0.30%) +1.18ms (+ 0.36%) 322.97ms 340.80ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey jakebailey changed the base branch from main to jabaile/tsc-services April 3, 2024 00:01
@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 3, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@jakebailey
Copy link
Member Author

Those results are a lot better than #57703 (comment).

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 288,737k (± 0.02%) 301,845k (± 0.01%) 🔻+13,108k (+ 4.54%) 301,820k 301,897k p=0.005 n=6
Parse Time 3.25s (± 0.42%) 3.24s (± 0.60%) ~ 3.22s 3.27s p=0.742 n=6
Bind Time 1.01s (± 1.36%) 0.99s (± 0.41%) -0.02s (- 2.30%) 0.99s 1.00s p=0.006 n=6
Check Time 10.64s (± 0.38%) 9.81s (± 0.18%) 🟩-0.83s (- 7.77%) 9.79s 9.84s p=0.005 n=6
Emit Time 9.44s (± 0.26%) 8.43s (± 0.32%) 🟩-1.02s (-10.75%) 8.39s 8.45s p=0.005 n=6
Total Time 24.34s (± 0.27%) 22.48s (± 0.21%) 🟩-1.87s (- 7.67%) 22.43s 22.55s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 195,077k (± 0.95%) 199,158k (± 0.78%) +4,081k (+ 2.09%) 197,450k 201,168k p=0.008 n=6
Parse Time 2.07s (± 1.27%) 2.07s (± 1.13%) ~ 2.03s 2.09s p=0.466 n=6
Bind Time 1.10s (± 0.50%) 1.09s (± 1.38%) ~ 1.07s 1.11s p=0.054 n=6
Check Time 15.14s (± 0.45%) 14.02s (± 0.77%) 🟩-1.12s (- 7.41%) 13.88s 14.20s p=0.005 n=6
Emit Time 4.29s (± 1.04%) 3.94s (± 5.00%) 🟩-0.35s (- 8.16%) 3.75s 4.13s p=0.005 n=6
Total Time 22.61s (± 0.50%) 21.12s (± 0.92%) 🟩-1.49s (- 6.58%) 20.94s 21.42s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 332,718k (± 0.01%) 353,354k (± 0.01%) 🔻+20,637k (+ 6.20%) 353,322k 353,417k p=0.005 n=6
Parse Time 3.57s (± 0.60%) 3.64s (± 2.99%) ~ 3.54s 3.85s p=0.107 n=6
Bind Time 1.72s (± 0.44%) 1.55s (± 6.46%) 🟩-0.17s (- 9.87%) 1.36s 1.61s p=0.004 n=6
Check Time 11.44s (± 0.27%) 10.18s (± 0.27%) 🟩-1.26s (-11.04%) 10.14s 10.21s p=0.005 n=6
Emit Time 6.68s (± 0.26%) 6.06s (± 0.45%) 🟩-0.62s (- 9.28%) 6.03s 6.11s p=0.005 n=6
Total Time 23.42s (± 0.22%) 21.43s (± 0.25%) 🟩-1.99s (- 8.48%) 21.36s 21.48s p=0.005 n=6
TFS - node (v18.15.0, x64)
Memory used 295,350k (± 0.08%) 308,515k (± 0.01%) 🔻+13,166k (+ 4.46%) 308,473k 308,548k p=0.005 n=6
Parse Time 2.00s (± 0.49%) 2.04s (± 0.88%) +0.04s (+ 1.92%) 2.02s 2.07s p=0.005 n=6
Bind Time 1.05s (± 0.72%) 1.03s (± 0.40%) -0.02s (- 1.90%) 1.03s 1.04s p=0.004 n=6
Check Time 7.25s (± 0.23%) 6.40s (± 0.43%) 🟩-0.85s (-11.74%) 6.37s 6.43s p=0.005 n=6
Emit Time 4.11s (± 0.45%) 3.60s (± 0.41%) 🟩-0.51s (-12.38%) 3.58s 3.61s p=0.005 n=6
Total Time 14.41s (± 0.25%) 13.07s (± 0.32%) 🟩-1.34s (- 9.30%) 13.01s 13.11s p=0.005 n=6
material-ui - node (v18.15.0, x64)
Memory used 488,586k (± 0.01%) 517,173k (± 0.01%) 🔻+28,587k (+ 5.85%) 517,120k 517,219k p=0.005 n=6
Parse Time 3.21s (± 0.75%) 3.25s (± 0.32%) +0.04s (+ 1.09%) 3.23s 3.26s p=0.048 n=6
Bind Time 1.27s (± 7.13%) 1.23s (± 1.26%) ~ 1.22s 1.26s p=0.063 n=6
Check Time 22.16s (± 0.41%) 20.62s (± 0.16%) 🟩-1.54s (- 6.95%) 20.58s 20.65s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 26.65s (± 0.32%) 25.09s (± 0.07%) 🟩-1.55s (- 5.82%) 25.07s 25.12s p=0.005 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,654,339k (± 0.00%) 1,750,659k (± 0.00%) 🔻+96,320k (+ 5.82%) 1,750,577k 1,750,724k p=0.005 n=6
Parse Time 9.59s (± 0.43%) 9.89s (± 1.77%) +0.30s (+ 3.18%) 9.76s 10.24s p=0.005 n=6
Bind Time 2.94s (± 0.88%) 2.79s (± 0.95%) 🟩-0.15s (- 5.04%) 2.76s 2.83s p=0.005 n=6
Check Time 91.27s (± 1.63%) 83.26s (± 1.14%) 🟩-8.01s (- 8.78%) 82.37s 84.57s p=0.005 n=6
Emit Time 0.19s (± 2.67%) 0.19s (± 2.81%) ~ 0.19s 0.20s p=0.640 n=6
Total Time 103.99s (± 1.42%) 96.14s (± 1.08%) 🟩-7.86s (- 7.55%) 95.21s 97.59s p=0.005 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,366,559k (± 0.12%) 2,490,452k (± 0.05%) 🔻+123,893k (+ 5.24%) 2,488,554k 2,492,257k p=0.005 n=6
Parse Time 7.67s (± 0.78%) 7.80s (± 1.04%) +0.13s (+ 1.67%) 7.67s 7.89s p=0.031 n=6
Bind Time 2.63s (± 0.51%) 2.52s (± 0.70%) 🟩-0.11s (- 4.31%) 2.50s 2.54s p=0.005 n=6
Check Time 53.33s (± 0.68%) 50.04s (± 0.17%) 🟩-3.29s (- 6.17%) 49.95s 50.20s p=0.005 n=6
Emit Time 4.42s (± 8.22%) 3.96s (± 0.65%) 🟩-0.46s (-10.45%) 3.92s 3.99s p=0.005 n=6
Total Time 68.05s (± 0.56%) 64.34s (± 0.23%) 🟩-3.71s (- 5.45%) 64.18s 64.55s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 406,697k (± 0.01%) 421,757k (± 0.01%) +15,060k (+ 3.70%) 421,728k 421,790k p=0.005 n=6
Parse Time 3.22s (± 0.61%) 3.24s (± 0.53%) ~ 3.22s 3.26s p=0.139 n=6
Bind Time 1.58s (± 0.67%) 1.43s (± 1.51%) 🟩-0.14s (- 8.89%) 1.41s 1.46s p=0.005 n=6
Check Time 19.50s (± 0.38%) 18.20s (± 0.26%) 🟩-1.30s (- 6.68%) 18.13s 18.27s p=0.005 n=6
Emit Time 1.37s (± 1.56%) 1.35s (± 1.66%) ~ 1.32s 1.38s p=0.332 n=6
Total Time 25.66s (± 0.34%) 24.22s (± 0.26%) 🟩-1.44s (- 5.60%) 24.14s 24.32s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,759,368k (± 0.00%) 2,901,769k (± 0.00%) 🔻+142,401k (+ 5.16%) 2,901,690k 2,901,833k p=0.005 n=6
Parse Time 11.03s (± 0.39%) 11.08s (± 0.49%) ~ 11.00s 11.14s p=0.125 n=6
Bind Time 3.62s (± 0.39%) 3.47s (± 0.50%) 🟩-0.15s (- 4.05%) 3.46s 3.50s p=0.005 n=6
Check Time 68.59s (± 0.52%) 62.22s (± 0.27%) 🟩-6.38s (- 9.30%) 61.92s 62.37s p=0.005 n=6
Emit Time 18.21s (± 0.35%) 16.59s (± 0.36%) 🟩-1.62s (- 8.87%) 16.54s 16.70s p=0.005 n=6
Total Time 101.45s (± 0.38%) 93.36s (± 0.17%) 🟩-8.09s (- 7.97%) 93.08s 93.50s p=0.005 n=6
webpack - node (v18.15.0, x64)
Memory used 396,884k (± 0.02%) 415,541k (± 0.01%) 🔻+18,657k (+ 4.70%) 415,460k 415,612k p=0.005 n=6
Parse Time 3.19s (± 1.63%) 3.31s (± 1.23%) +0.11s (+ 3.60%) 3.26s 3.38s p=0.008 n=6
Bind Time 1.47s (± 0.35%) 1.38s (± 0.40%) 🟩-0.08s (- 5.57%) 1.38s 1.39s p=0.004 n=6
Check Time 15.29s (± 0.24%) 14.45s (± 0.32%) 🟩-0.85s (- 5.53%) 14.36s 14.48s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 19.96s (± 0.19%) 19.14s (± 0.42%) 🟩-0.81s (- 4.08%) 19.00s 19.24s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 494,548k (± 0.03%) 531,647k (± 0.02%) 🔻+37,099k (+ 7.50%) 531,487k 531,805k p=0.005 n=6
Parse Time 4.93s (± 0.25%) 4.96s (± 0.52%) +0.03s (+ 0.57%) 4.94s 5.01s p=0.029 n=6
Bind Time 2.21s (± 1.20%) 2.08s (± 0.83%) 🟩-0.14s (- 6.10%) 2.05s 2.10s p=0.005 n=6
Check Time 4.68s (± 1.04%) 4.48s (± 0.85%) 🟩-0.20s (- 4.20%) 4.44s 4.53s p=0.005 n=6
Emit Time 0.11s (± 0.00%) 0.10s (± 0.00%) 🟩-0.01s (- 9.09%) 0.10s 0.10s p=0.001 n=6
Total Time 11.94s (± 0.54%) 11.63s (± 0.50%) -0.31s (- 2.60%) 11.54s 11.68s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,369ms (± 0.49%) 2,369ms (± 0.88%) ~ 2,335ms 2,395ms p=0.873 n=6
Req 2 - geterr 5,629ms (± 0.43%) 5,131ms (± 0.55%) 🟩-498ms (- 8.85%) 5,078ms 5,161ms p=0.005 n=6
Req 3 - references 318ms (± 0.47%) 311ms (± 1.42%) -7ms (- 2.20%) 308ms 317ms p=0.011 n=6
Req 4 - navto 271ms (± 0.60%) 269ms (± 3.64%) -2ms (- 0.86%) 265ms 289ms p=0.047 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 89ms (± 4.97%) 77ms (± 0.72%) 🟩-13ms (-14.04%) 76ms 77ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,721ms (± 0.68%) 3,672ms (± 0.93%) ~ 3,643ms 3,734ms p=0.054 n=6
Req 2 - geterr 6,110ms (± 0.27%) 5,647ms (± 0.51%) 🟩-463ms (- 7.57%) 5,598ms 5,682ms p=0.005 n=6
Req 3 - references 494ms (± 0.30%) 465ms (± 0.46%) 🟩-29ms (- 5.77%) 462ms 468ms p=0.005 n=6
Req 4 - navto 446ms (± 0.63%) 415ms (± 1.76%) 🟩-31ms (- 6.85%) 402ms 422ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 108ms (± 0.95%) 109ms (± 0.95%) ~ 107ms 110ms p=0.652 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,351ms (±10.16%) 3,096ms (± 0.58%) 🟩-255ms (- 7.61%) 3,076ms 3,123ms p=0.013 n=6
Req 2 - geterr 2,380ms (± 5.52%) 2,127ms (±10.43%) 🟩-254ms (-10.66%) 1,832ms 2,280ms p=0.031 n=6
Req 3 - references 149ms (± 1.84%) 139ms (± 1.56%) 🟩-10ms (- 6.62%) 136ms 142ms p=0.005 n=6
Req 4 - navto 552ms (± 0.34%) 537ms (± 1.30%) -15ms (- 2.75%) 532ms 551ms p=0.010 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 455ms (± 1.87%) 443ms (± 0.84%) -12ms (- 2.67%) 438ms 449ms p=0.015 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 228.74ms (± 0.15%) 228.80ms (± 0.14%) +0.06ms (+ 0.03%) 227.49ms 233.92ms p=0.014 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.75ms (± 0.15%) 227.71ms (± 0.21%) -0.04ms (- 0.02%) 226.21ms 236.35ms p=0.013 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 223.62ms (± 0.16%) 223.63ms (± 0.17%) ~ 221.93ms 227.33ms p=0.993 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.46ms (± 0.15%) 223.55ms (± 0.16%) +0.09ms (+ 0.04%) 221.95ms 225.83ms p=0.021 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Yessss

@jakebailey

This comment was marked as outdated.

Sign in to view
@typescript-bot

This comment was marked as outdated.

Sign in to view
@typescript-bot

This comment was marked as outdated.

Sign in to view
@jakebailey

This comment was marked as outdated.

Sign in to view
@jakebailey

This comment was marked as outdated.

Sign in to view
@jakebailey
Copy link
Member Author

FWIW comparing #51682 (comment), this increase memory is not unexpected. Maybe it's fine; this benchmark is already extra strange because I'm emitting stuff, which won't happen in tsserver... normally anyway.

I feel like if #51682 (comment) was "good", then this PR is good.

@jakebailey jakebailey mentioned this pull request Apr 3, 2024
Merged
@jakebailey jakebailey requested a review from rbuckton April 3, 2024 17:42
@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 3, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 288,752k (± 0.01%) 305,167k (± 0.01%) 🔻+16,415k (+ 5.68%) 305,144k 305,200k p=0.005 n=6
Parse Time 4.01s (± 0.49%) 4.03s (± 0.41%) ~ 4.00s 4.05s p=0.406 n=6
Bind Time 1.27s (± 0.77%) 1.25s (± 0.98%) -0.02s (- 1.84%) 1.23s 1.26s p=0.008 n=6
Check Time 13.19s (± 0.41%) 12.15s (± 0.67%) 🟩-1.04s (- 7.92%) 12.05s 12.24s p=0.005 n=6
Emit Time 11.76s (± 0.35%) 10.54s (± 0.53%) 🟩-1.23s (-10.42%) 10.48s 10.62s p=0.005 n=6
Total Time 30.24s (± 0.26%) 27.95s (± 0.17%) 🟩-2.29s (- 7.56%) 27.87s 28.00s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 195,094k (± 0.95%) 201,384k (± 0.97%) +6,290k (+ 3.22%) 198,954k 203,597k p=0.005 n=6
Parse Time 2.04s (± 1.15%) 2.09s (± 0.56%) +0.05s (+ 2.28%) 2.07s 2.10s p=0.011 n=6
Bind Time 1.11s (± 0.73%) 1.08s (± 1.23%) -0.03s (- 2.84%) 1.06s 1.10s p=0.006 n=6
Check Time 15.17s (± 0.66%) 13.97s (± 0.36%) 🟩-1.19s (- 7.88%) 13.93s 14.07s p=0.005 n=6
Emit Time 4.33s (± 0.92%) 3.83s (± 3.45%) 🟩-0.49s (-11.40%) 3.74s 4.10s p=0.005 n=6
Total Time 22.65s (± 0.44%) 20.97s (± 0.57%) 🟩-1.67s (- 7.40%) 20.89s 21.21s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 332,724k (± 0.01%) 357,725k (± 0.01%) 🔻+25,001k (+ 7.51%) 357,705k 357,749k p=0.005 n=6
Parse Time 3.53s (± 1.17%) 3.70s (± 2.86%) 🔻+0.17s (+ 4.91%) 3.52s 3.81s p=0.020 n=6
Bind Time 1.71s (± 0.71%) 1.42s (± 6.28%) 🟩-0.30s (-17.41%) 1.33s 1.56s p=0.005 n=6
Check Time 11.41s (± 0.35%) 10.21s (± 0.52%) 🟩-1.20s (-10.51%) 10.14s 10.29s p=0.005 n=6
Emit Time 6.69s (± 0.31%) 6.06s (± 0.55%) 🟩-0.63s (- 9.44%) 6.00s 6.10s p=0.005 n=6
Total Time 23.34s (± 0.20%) 21.38s (± 0.31%) 🟩-1.96s (- 8.39%) 21.29s 21.46s p=0.005 n=6
TFS - node (v18.15.0, x64)
Memory used 295,145k (± 0.11%) 311,915k (± 0.00%) 🔻+16,769k (+ 5.68%) 311,903k 311,927k p=0.005 n=6
Parse Time 2.01s (± 0.80%) 2.04s (± 0.67%) +0.02s (+ 1.24%) 2.01s 2.05s p=0.031 n=6
Bind Time 1.06s (± 0.71%) 1.03s (± 1.19%) 🟩-0.04s (- 3.45%) 1.00s 1.03s p=0.003 n=6
Check Time 7.25s (± 0.27%) 6.39s (± 0.38%) 🟩-0.85s (-11.75%) 6.37s 6.42s p=0.005 n=6
Emit Time 4.10s (± 0.51%) 3.60s (± 0.44%) 🟩-0.50s (-12.24%) 3.58s 3.62s p=0.005 n=6
Total Time 14.42s (± 0.21%) 13.06s (± 0.34%) 🟩-1.36s (- 9.44%) 13.01s 13.12s p=0.005 n=6
material-ui - node (v18.15.0, x64)
Memory used 488,567k (± 0.00%) 519,876k (± 0.01%) 🔻+31,309k (+ 6.41%) 519,811k 519,920k p=0.005 n=6
Parse Time 3.20s (± 0.80%) 3.28s (± 1.45%) +0.08s (+ 2.50%) 3.22s 3.36s p=0.010 n=6
Bind Time 1.16s (± 6.63%) 1.22s (± 1.61%) ~ 1.18s 1.23s p=0.063 n=6
Check Time 22.14s (± 0.30%) 20.62s (± 0.44%) 🟩-1.52s (- 6.84%) 20.52s 20.73s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 26.49s (± 0.46%) 25.11s (± 0.36%) 🟩-1.37s (- 5.18%) 24.99s 25.20s p=0.005 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,654,303k (± 0.00%) 1,761,800k (± 0.00%) 🔻+107,497k (+ 6.50%) 1,761,758k 1,761,873k p=0.005 n=6
Parse Time 9.62s (± 0.38%) 9.85s (± 0.57%) +0.22s (+ 2.34%) 9.79s 9.95s p=0.005 n=6
Bind Time 2.97s (± 0.66%) 2.83s (± 0.65%) 🟩-0.14s (- 4.55%) 2.81s 2.85s p=0.005 n=6
Check Time 91.22s (± 1.10%) 82.52s (± 0.71%) 🟩-8.70s (- 9.53%) 81.75s 83.17s p=0.005 n=6
Emit Time 0.19s (± 4.29%) 0.20s (± 3.79%) ~ 0.19s 0.21s p=0.432 n=6
Total Time 104.00s (± 0.98%) 95.40s (± 0.63%) 🟩-8.60s (- 8.27%) 94.57s 96.05s p=0.005 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,366,974k (± 0.14%) 2,510,448k (± 0.02%) 🔻+143,474k (+ 6.06%) 2,509,934k 2,511,035k p=0.005 n=6
Parse Time 6.17s (± 0.73%) 6.26s (± 1.20%) +0.09s (+ 1.49%) 6.17s 6.34s p=0.045 n=6
Bind Time 2.13s (± 0.92%) 2.07s (± 1.00%) -0.06s (- 2.90%) 2.04s 2.09s p=0.005 n=6
Check Time 43.04s (± 0.51%) 40.48s (± 0.27%) 🟩-2.56s (- 5.96%) 40.34s 40.64s p=0.005 n=6
Emit Time 3.56s (± 8.31%) 3.18s (± 1.20%) 🟩-0.38s (-10.76%) 3.13s 3.24s p=0.008 n=6
Total Time 54.91s (± 0.66%) 51.98s (± 0.27%) 🟩-2.92s (- 5.32%) 51.83s 52.22s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 406,636k (± 0.01%) 425,786k (± 0.01%) 🔻+19,151k (+ 4.71%) 425,738k 425,822k p=0.005 n=6
Parse Time 2.67s (± 0.77%) 2.84s (± 0.47%) 🔻+0.17s (+ 6.23%) 2.82s 2.86s p=0.005 n=6
Bind Time 1.33s (± 0.63%) 1.08s (± 0.38%) 🟩-0.25s (-18.62%) 1.07s 1.08s p=0.003 n=6
Check Time 16.52s (± 0.63%) 15.42s (± 0.44%) 🟩-1.10s (- 6.68%) 15.33s 15.51s p=0.005 n=6
Emit Time 1.15s (± 1.31%) 1.13s (± 1.21%) -0.02s (- 1.74%) 1.11s 1.15s p=0.040 n=6
Total Time 21.67s (± 0.48%) 20.46s (± 0.37%) 🟩-1.21s (- 5.57%) 20.36s 20.55s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,760,260k (± 0.00%) 2,930,540k (± 0.00%) 🔻+170,280k (+ 6.17%) 2,930,498k 2,930,611k p=0.005 n=6
Parse Time 10.99s (± 0.39%) 11.12s (± 0.64%) +0.13s (+ 1.17%) 11.02s 11.21s p=0.013 n=6
Bind Time 3.61s (± 0.37%) 3.49s (± 0.16%) 🟩-0.12s (- 3.23%) 3.49s 3.50s p=0.004 n=6
Check Time 68.53s (± 0.34%) 62.26s (± 0.40%) 🟩-6.27s (- 9.14%) 61.91s 62.57s p=0.005 n=6
Emit Time 18.26s (± 0.35%) 16.62s (± 0.85%) 🟩-1.64s (- 8.98%) 16.45s 16.83s p=0.005 n=6
Total Time 101.39s (± 0.27%) 93.49s (± 0.44%) 🟩-7.90s (- 7.79%) 92.97s 94.11s p=0.005 n=6
webpack - node (v18.15.0, x64)
Memory used 396,897k (± 0.03%) 419,139k (± 0.02%) 🔻+22,241k (+ 5.60%) 419,062k 419,241k p=0.005 n=6
Parse Time 3.19s (± 2.45%) 3.34s (± 0.76%) 🔻+0.15s (+ 4.76%) 3.32s 3.39s p=0.014 n=6
Bind Time 1.46s (± 0.87%) 1.39s (± 0.98%) 🟩-0.07s (- 5.02%) 1.37s 1.40s p=0.005 n=6
Check Time 15.37s (± 0.23%) 14.45s (± 0.38%) 🟩-0.92s (- 5.95%) 14.39s 14.54s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.02s (± 0.43%) 19.18s (± 0.28%) 🟩-0.84s (- 4.17%) 19.12s 19.24s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 494,503k (± 0.01%) 537,535k (± 0.02%) 🔻+43,032k (+ 8.70%) 537,399k 537,717k p=0.005 n=6
Parse Time 4.91s (± 0.60%) 4.96s (± 0.42%) +0.04s (+ 0.81%) 4.93s 4.98s p=0.029 n=6
Bind Time 2.25s (± 0.95%) 2.21s (± 0.55%) -0.04s (- 1.56%) 2.20s 2.23s p=0.010 n=6
Check Time 4.69s (± 0.57%) 4.43s (± 1.00%) 🟩-0.26s (- 5.54%) 4.38s 4.50s p=0.005 n=6
Emit Time 0.11s (± 0.00%) 0.10s (± 4.01%) 🟩-0.01s (- 7.58%) 0.10s 0.11s p=0.007 n=6
Total Time 11.96s (± 0.29%) 11.71s (± 0.49%) -0.25s (- 2.12%) 11.66s 11.81s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,556ms (± 0.34%) 3,532ms (± 0.52%) -25ms (- 0.70%) 3,511ms 3,563ms p=0.045 n=6
Req 2 - geterr 8,310ms (± 0.37%) 7,691ms (± 2.17%) 🟩-618ms (- 7.44%) 7,597ms 8,029ms p=0.005 n=6
Req 3 - references 476ms (± 0.91%) 428ms (± 1.93%) 🟩-48ms (-10.08%) 421ms 440ms p=0.005 n=6
Req 4 - navto 406ms (± 0.59%) 343ms (± 0.80%) 🟩-63ms (-15.61%) 337ms 344ms p=0.004 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 132ms (± 5.18%) 121ms (± 8.64%) 🟩-12ms (- 8.94%) 109ms 138ms p=0.045 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,223ms (±11.83%) 3,065ms (± 9.62%) ~ 2,899ms 3,665ms p=0.092 n=6
Req 2 - geterr 5,515ms (±11.66%) 4,782ms (± 8.77%) 🟩-733ms (-13.29%) 4,571ms 5,637ms p=0.020 n=6
Req 3 - references 450ms (± 8.57%) 419ms (± 9.57%) ~ 366ms 448ms p=0.810 n=6
Req 4 - navto 443ms (± 1.57%) 320ms (± 8.48%) 🟩-123ms (-27.79%) 278ms 342ms p=0.005 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 109ms (± 1.13%) 95ms (±10.14%) 🟩-14ms (-12.60%) 82ms 102ms p=0.004 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,635ms (± 0.43%) 2,589ms (± 0.33%) -47ms (- 1.77%) 2,575ms 2,598ms p=0.005 n=6
Req 2 - geterr 1,753ms (± 1.84%) 1,545ms (± 0.66%) 🟩-208ms (-11.87%) 1,535ms 1,558ms p=0.005 n=6
Req 3 - references 115ms (± 9.25%) 94ms (± 0.80%) 🟩-21ms (-17.88%) 93ms 95ms p=0.005 n=6
Req 4 - navto 368ms (± 0.32%) 356ms (± 1.30%) 🟩-12ms (- 3.26%) 351ms 363ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 311ms (± 1.69%) 290ms (± 1.35%) 🟩-21ms (- 6.66%) 284ms 296ms p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 228.18ms (± 0.17%) 228.13ms (± 0.14%) ~ 226.56ms 230.84ms p=0.660 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.66ms (± 0.15%) 227.55ms (± 0.19%) -0.10ms (- 0.05%) 226.04ms 234.01ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 330.90ms (± 0.30%) 330.90ms (± 0.29%) ~ 322.58ms 339.22ms p=0.839 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 331.13ms (± 0.32%) 330.93ms (± 0.29%) -0.20ms (- 0.06%) 322.95ms 340.40ms p=0.005 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

The _children change appears to have even more greatly improved tsserver performance. Awesome.

@jakebailey jakebailey changed the base branch from jabaile/tsc-services to main April 4, 2024 18:24
@typescript-bot
Copy link
Collaborator

Looks like you're introducing a change to the public API surface area. If this includes breaking changes, please document them on our wiki's API Breaking Changes page.

Also, please make sure @DanielRosenwasser and @RyanCavanaugh are aware of the changes, just as a heads up.

@jakebailey
Copy link
Member Author

I've rebased and retargeted the PR to main. Rerunning benchmarks to check without forcing tsc's allocators.

@typescript-bot perf test this
@typescript-bot run dt
@typescript-bot user test this
@typescript-bot test top400

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 4, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results
run dt ✅ Started ✅ Results
user test this ✅ Started ✅ Results
test top400 ✅ Started ✅ Results

@jakebailey
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 4, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
pack this ✅ Started ✅ Results

@typescript-bot
Copy link
Collaborator

Hey @jakebailey, the results of running the DT tests are ready.

Everything looks the same!

You can check the log here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 4, 2024 •
edited
Loading

Hey @jakebailey, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/160982/artifacts?artifactName=tgz&fileId=B08E5D89B7EC46EFFAA2687CDAB8467544AC7B5B9F1217BCA7F8AEA8941E259E02&fileName=/typescript-5.5.0-insiders.20240404.tgz"
    }
}

and then running npm install.

There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the user tests comparing main and refs/pull/58045/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,661k (± 0.02%) 295,654k (± 0.01%) ~ 295,616k 295,680k p=0.936 n=6
Parse Time 3.93s (± 0.41%) 3.93s (± 0.39%) ~ 3.91s 3.95s p=0.739 n=6
Bind Time 1.23s (± 1.49%) 1.23s (± 0.95%) ~ 1.22s 1.25s p=0.731 n=6
Check Time 11.99s (± 0.36%) 12.05s (± 0.16%) +0.06s (+ 0.47%) 12.03s 12.08s p=0.013 n=6
Emit Time 10.47s (± 0.36%) 10.47s (± 0.20%) ~ 10.45s 10.51s p=0.809 n=6
Total Time 27.62s (± 0.15%) 27.68s (± 0.14%) +0.06s (+ 0.21%) 27.61s 27.72s p=0.037 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,950k (± 1.02%) 192,748k (± 0.74%) ~ 192,112k 195,649k p=0.575 n=6
Parse Time 2.02s (± 0.93%) 2.00s (± 1.72%) ~ 1.95s 2.05s p=0.687 n=6
Bind Time 1.08s (± 0.95%) 1.08s (± 0.59%) ~ 1.07s 1.09s p=0.443 n=6
Check Time 13.92s (± 0.30%) 13.88s (± 0.30%) ~ 13.83s 13.94s p=0.225 n=6
Emit Time 3.84s (± 0.45%) 3.85s (± 0.85%) ~ 3.81s 3.89s p=0.936 n=6
Total Time 20.86s (± 0.22%) 20.81s (± 0.37%) ~ 20.74s 20.93s p=0.294 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,539k (± 0.01%) 347,541k (± 0.01%) ~ 347,527k 347,573k p=1.000 n=6
Parse Time 3.01s (± 0.93%) 2.97s (± 0.84%) ~ 2.94s 3.01s p=0.062 n=6
Bind Time 1.12s (± 0.46%) 1.11s (± 0.46%) ~ 1.11s 1.12s p=0.311 n=6
Check Time 8.26s (± 0.45%) 8.27s (± 0.25%) ~ 8.23s 8.29s p=0.629 n=6
Emit Time 4.87s (± 0.64%) 4.84s (± 0.35%) ~ 4.82s 4.86s p=0.146 n=6
Total Time 17.25s (± 0.40%) 17.19s (± 0.19%) ~ 17.16s 17.25s p=0.126 n=6
TFS - node (v18.15.0, x64)
Memory used 302,832k (± 0.00%) 302,852k (± 0.01%) ~ 302,817k 302,876k p=0.148 n=6
Parse Time 2.96s (± 0.87%) 2.96s (± 0.70%) ~ 2.93s 2.99s p=0.806 n=6
Bind Time 1.48s (± 0.28%) 1.48s (± 0.82%) ~ 1.47s 1.50s p=0.116 n=6
Check Time 9.25s (± 0.23%) 9.26s (± 0.33%) ~ 9.22s 9.31s p=0.627 n=6
Emit Time 5.31s (± 0.31%) 5.31s (± 0.60%) ~ 5.28s 5.36s p=0.570 n=6
Total Time 19.00s (± 0.21%) 19.01s (± 0.18%) ~ 18.96s 19.05s p=0.872 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,182k (± 0.01%) 510,238k (± 0.00%) ~ 510,208k 510,264k p=0.078 n=6
Parse Time 3.94s (± 0.30%) 3.95s (± 0.28%) ~ 3.94s 3.96s p=0.354 n=6
Bind Time 1.46s (± 0.71%) 1.47s (± 0.82%) ~ 1.45s 1.48s p=0.210 n=6
Check Time 25.46s (± 0.25%) 25.40s (± 0.36%) ~ 25.29s 25.53s p=0.226 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.86s (± 0.21%) 30.82s (± 0.30%) ~ 30.71s 30.96s p=0.520 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,929k (± 0.00%) 1,744,950k (± 0.00%) ~ 1,744,902k 1,745,029k p=0.378 n=6
Parse Time 9.65s (± 0.50%) 9.63s (± 0.59%) ~ 9.58s 9.74s p=0.419 n=6
Bind Time 3.49s (± 0.23%) 3.48s (± 0.63%) ~ 3.45s 3.51s p=0.461 n=6
Check Time 82.32s (± 0.39%) 82.24s (± 0.53%) ~ 81.89s 82.94s p=0.748 n=6
Emit Time 0.19s (± 2.13%) 0.19s (± 2.13%) ~ 0.19s 0.20s p=1.000 n=6
Total Time 95.64s (± 0.35%) 95.54s (± 0.50%) ~ 95.13s 96.34s p=0.471 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,402,045k (± 0.05%) 2,402,269k (± 0.03%) ~ 2,401,211k 2,402,807k p=0.689 n=6
Parse Time 7.48s (± 0.77%) 7.54s (± 0.83%) ~ 7.46s 7.62s p=0.173 n=6
Bind Time 2.79s (± 0.57%) 2.79s (± 0.88%) ~ 2.75s 2.82s p=1.000 n=6
Check Time 49.38s (± 0.18%) 49.19s (± 0.30%) -0.19s (- 0.38%) 49.02s 49.41s p=0.031 n=6
Emit Time 3.99s (± 3.27%) 3.90s (± 1.67%) -0.09s (- 2.38%) 3.86s 4.03s p=0.044 n=6
Total Time 63.64s (± 0.15%) 63.44s (± 0.22%) -0.20s (- 0.31%) 63.30s 63.66s p=0.045 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,584k (± 0.01%) 416,592k (± 0.01%) ~ 416,536k 416,654k p=1.000 n=6
Parse Time 4.11s (± 0.42%) 4.12s (± 0.40%) ~ 4.10s 4.14s p=0.684 n=6
Bind Time 1.59s (± 0.84%) 1.59s (± 0.62%) ~ 1.58s 1.60s p=0.727 n=6
Check Time 22.45s (± 0.49%) 22.36s (± 0.38%) ~ 22.27s 22.49s p=0.199 n=6
Emit Time 1.66s (± 2.72%) 1.68s (± 1.02%) ~ 1.66s 1.71s p=0.414 n=6
Total Time 29.81s (± 0.49%) 29.75s (± 0.32%) ~ 29.66s 29.91s p=0.334 n=6
vscode - node (v18.15.0, x64)
Memory used 2,907,964k (± 0.00%) 2,907,969k (± 0.00%) ~ 2,907,922k 2,908,007k p=0.810 n=6
Parse Time 10.84s (± 0.29%) 10.84s (± 0.15%) ~ 10.82s 10.86s p=0.808 n=6
Bind Time 3.49s (± 0.38%) 3.49s (± 0.26%) ~ 3.48s 3.50s p=0.932 n=6
Check Time 62.16s (± 0.46%) 62.22s (± 0.49%) ~ 61.79s 62.66s p=0.748 n=6
Emit Time 16.49s (± 0.47%) 16.43s (± 0.41%) ~ 16.32s 16.51s p=0.173 n=6
Total Time 92.98s (± 0.25%) 92.98s (± 0.26%) ~ 92.64s 93.31s p=1.000 n=6
webpack - node (v18.15.0, x64)
Memory used 408,998k (± 0.02%) 408,982k (± 0.03%) ~ 408,858k 409,216k p=0.575 n=6
Parse Time 3.89s (± 0.50%) 3.89s (± 0.66%) ~ 3.87s 3.94s p=0.871 n=6
Bind Time 1.69s (± 0.95%) 1.68s (± 0.75%) ~ 1.66s 1.69s p=0.735 n=6
Check Time 16.86s (± 0.32%) 16.83s (± 0.22%) ~ 16.76s 16.86s p=0.374 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.44s (± 0.14%) 22.40s (± 0.26%) ~ 22.29s 22.45s p=0.227 n=6
xstate - node (v18.15.0, x64)
Memory used 513,425k (± 0.01%) 513,449k (± 0.00%) ~ 513,418k 513,474k p=0.810 n=6
Parse Time 3.28s (± 0.31%) 3.29s (± 0.32%) ~ 3.27s 3.30s p=0.801 n=6
Bind Time 1.57s (± 0.48%) 1.57s (± 0.33%) ~ 1.57s 1.58s p=0.241 n=6
Check Time 2.91s (± 0.36%) 2.90s (± 0.77%) ~ 2.87s 2.92s p=0.405 n=6
Emit Time 0.07s (±11.12%) 0.07s (±11.12%) ~ 0.07s 0.09s p=1.000 n=6
Total Time 7.85s (± 0.15%) 7.83s (± 0.21%) ~ 7.81s 7.85s p=0.224 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,377ms (± 0.93%) 2,345ms (± 0.34%) -32ms (- 1.34%) 2,333ms 2,355ms p=0.008 n=6
Req 2 - geterr 5,639ms (± 0.34%) 5,150ms (± 0.59%) 🟩-490ms (- 8.69%) 5,117ms 5,206ms p=0.005 n=6
Req 3 - references 319ms (± 1.04%) 295ms (± 2.10%) 🟩-24ms (- 7.38%) 287ms 299ms p=0.004 n=6
Req 4 - navto 272ms (± 0.00%) 228ms (± 0.00%) 🟩-44ms (-16.18%) 228ms 228ms p=0.001 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 88ms (± 2.32%) 77ms (± 0.67%) 🟩-11ms (-11.95%) 77ms 78ms p=0.003 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,520ms (± 0.73%) 2,509ms (± 0.89%) ~ 2,476ms 2,537ms p=0.471 n=6
Req 2 - geterr 4,159ms (± 0.20%) 3,900ms (± 1.53%) 🟩-259ms (- 6.23%) 3,863ms 4,020ms p=0.005 n=6
Req 3 - references 333ms (± 0.36%) 298ms (± 0.87%) 🟩-35ms (-10.52%) 294ms 301ms p=0.005 n=6
Req 4 - navto 298ms (± 1.07%) 237ms (± 3.54%) 🟩-61ms (-20.57%) 226ms 243ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 75ms (± 5.90%) 70ms (± 3.06%) 🟩-5ms (- 6.89%) 68ms 74ms p=0.032 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,896ms (± 0.37%) 3,809ms (± 0.30%) -88ms (- 2.25%) 3,788ms 3,820ms p=0.005 n=6
Req 2 - geterr 2,450ms (± 0.98%) 2,279ms (± 1.02%) 🟩-171ms (- 6.97%) 2,255ms 2,314ms p=0.005 n=6
Req 3 - references 149ms (± 1.02%) 134ms (± 1.54%) 🟩-14ms (- 9.54%) 131ms 136ms p=0.005 n=6
Req 4 - navto 561ms (± 1.30%) 524ms (± 0.58%) 🟩-37ms (- 6.51%) 520ms 529ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 458ms (± 1.99%) 442ms (± 0.75%) 🟩-16ms (- 3.46%) 436ms 446ms p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.94ms (± 0.17%) 153.78ms (± 0.18%) -0.16ms (- 0.11%) 152.76ms 157.53ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.11ms (± 0.14%) 228.14ms (± 0.18%) ~ 226.50ms 233.04ms p=0.942 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 223.40ms (± 0.14%) 223.43ms (± 0.16%) ~ 221.96ms 228.18ms p=0.740 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.20ms (± 0.16%) 223.37ms (± 0.18%) +0.17ms (+ 0.08%) 221.68ms 229.09ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

The above looks exactly as expected for this PR targeting main; no change to tsc (it doesn't use this code), and all benefit for tsserver.

@jakebailey jakebailey mentioned this pull request Apr 4, 2024
Closed
@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the top 400 repos comparing main and refs/pull/58045/merge:

Everything looks good!

@jakebailey jakebailey changed the title Monomorphize services allocators, just like the compiler Monomorphize allocators for tsserver/public API, just like core compiler Apr 4, 2024
@jakebailey
Copy link
Member Author

The benchmarking system now includes a benchmark for simulating tsc via the public API; testing:

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 4, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,691k (± 0.01%) 295,683k (± 0.01%) ~ 295,657k 295,732k p=1.000 n=6
Parse Time 3.93s (± 0.21%) 3.94s (± 0.41%) ~ 3.92s 3.96s p=0.392 n=6
Bind Time 1.23s (± 0.84%) 1.23s (± 0.51%) ~ 1.22s 1.24s p=0.654 n=6
Check Time 12.04s (± 0.17%) 12.02s (± 0.28%) ~ 11.98s 12.06s p=0.513 n=6
Emit Time 10.46s (± 0.34%) 10.46s (± 0.44%) ~ 10.41s 10.53s p=1.000 n=6
Total Time 27.65s (± 0.09%) 27.64s (± 0.28%) ~ 27.56s 27.74s p=0.573 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,766k (± 0.76%) 193,374k (± 0.92%) ~ 192,059k 195,701k p=0.689 n=6
Parse Time 2.04s (± 1.05%) 2.02s (± 1.50%) ~ 1.98s 2.07s p=0.139 n=6
Bind Time 1.08s (± 0.59%) 1.06s (± 0.49%) -0.02s (- 1.54%) 1.06s 1.07s p=0.006 n=6
Check Time 13.89s (± 0.60%) 13.96s (± 0.49%) ~ 13.88s 14.06s p=0.199 n=6
Emit Time 3.85s (± 0.76%) 3.86s (± 0.45%) ~ 3.84s 3.89s p=0.565 n=6
Total Time 20.86s (± 0.37%) 20.91s (± 0.38%) ~ 20.83s 21.05s p=0.521 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,540k (± 0.01%) 347,538k (± 0.01%) ~ 347,504k 347,582k p=0.630 n=6
Parse Time 3.68s (± 1.47%) 3.69s (± 1.11%) ~ 3.63s 3.74s p=0.936 n=6
Bind Time 1.37s (± 1.10%) 1.39s (± 1.90%) ~ 1.37s 1.44s p=0.217 n=6
Check Time 10.17s (± 0.43%) 10.20s (± 0.46%) ~ 10.14s 10.27s p=0.258 n=6
Emit Time 6.01s (± 0.46%) 6.00s (± 0.42%) ~ 5.98s 6.05s p=0.806 n=6
Total Time 21.23s (± 0.31%) 21.28s (± 0.23%) ~ 21.22s 21.35s p=0.198 n=6
TFS - node (v18.15.0, x64)
Memory used 302,846k (± 0.01%) 302,835k (± 0.00%) ~ 302,821k 302,842k p=0.471 n=6
Parse Time 2.97s (± 0.72%) 2.97s (± 1.16%) ~ 2.94s 3.01s p=0.462 n=6
Bind Time 1.48s (± 0.35%) 1.47s (± 0.51%) ~ 1.46s 1.48s p=0.247 n=6
Check Time 9.26s (± 0.25%) 9.27s (± 0.26%) ~ 9.24s 9.31s p=0.627 n=6
Emit Time 5.30s (± 0.45%) 5.30s (± 0.94%) ~ 5.25s 5.38s p=1.000 n=6
Total Time 19.00s (± 0.21%) 19.01s (± 0.23%) ~ 18.95s 19.08s p=0.807 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,160k (± 0.01%) 510,166k (± 0.01%) ~ 510,112k 510,278k p=1.000 n=6
Parse Time 3.92s (± 0.36%) 3.93s (± 0.30%) ~ 3.92s 3.95s p=0.357 n=6
Bind Time 1.45s (± 0.71%) 1.46s (± 0.80%) ~ 1.45s 1.48s p=0.054 n=6
Check Time 25.48s (± 0.21%) 25.43s (± 0.43%) ~ 25.30s 25.57s p=0.689 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.84s (± 0.17%) 30.82s (± 0.37%) ~ 30.67s 30.95s p=1.000 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,935k (± 0.00%) 1,744,950k (± 0.00%) ~ 1,744,913k 1,745,007k p=0.689 n=6
Parse Time 9.59s (± 0.41%) 9.64s (± 0.50%) ~ 9.59s 9.72s p=0.149 n=6
Bind Time 3.45s (± 0.70%) 3.45s (± 1.08%) ~ 3.39s 3.49s p=0.685 n=6
Check Time 81.68s (± 0.30%) 82.20s (± 0.34%) +0.52s (+ 0.63%) 81.70s 82.48s p=0.025 n=6
Emit Time 0.19s (± 2.13%) 0.19s (± 2.67%) ~ 0.19s 0.20s p=0.595 n=6
Total Time 94.91s (± 0.28%) 95.48s (± 0.27%) +0.57s (+ 0.60%) 95.06s 95.78s p=0.013 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,296,646k (± 0.03%) 2,296,712k (± 0.03%) ~ 2,296,068k 2,297,638k p=1.000 n=6
Parse Time 7.42s (± 0.91%) 7.43s (± 1.15%) ~ 7.30s 7.54s p=0.810 n=6
Bind Time 2.78s (± 0.95%) 2.82s (± 1.08%) +0.04s (+ 1.44%) 2.78s 2.87s p=0.045 n=6
Check Time 49.34s (± 0.19%) 49.40s (± 0.25%) ~ 49.20s 49.55s p=0.298 n=6
Emit Time 3.98s (± 0.63%) 4.04s (± 3.47%) ~ 3.92s 4.29s p=1.000 n=6
Total Time 63.50s (± 0.21%) 63.70s (± 0.34%) ~ 63.35s 63.98s p=0.093 n=6
self-build-src-public-api - node (v18.15.0, x64)
Memory used 2,257,958k (± 0.03%) 2,371,367k (± 0.02%) 🔻+113,409k (+ 5.02%) 2,370,644k 2,372,032k p=0.005 n=6
Parse Time 5.08s (± 1.65%) 5.17s (± 0.49%) ~ 5.14s 5.20s p=0.093 n=6
Bind Time 1.80s (± 1.04%) 1.70s (± 1.22%) 🟩-0.10s (- 5.66%) 1.67s 1.72s p=0.005 n=6
Check Time 36.50s (± 0.40%) 34.36s (± 0.27%) 🟩-2.14s (- 5.87%) 34.27s 34.50s p=0.005 n=6
Emit Time 2.93s (± 1.41%) 2.69s (± 0.94%) 🟩-0.24s (- 8.09%) 2.66s 2.72s p=0.005 n=6
Total Time 46.32s (± 0.32%) 43.94s (± 0.17%) 🟩-2.38s (- 5.13%) 43.84s 44.05s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 419,494k (± 0.01%) 419,506k (± 0.01%) ~ 419,464k 419,559k p=0.575 n=6
Parse Time 4.11s (± 0.71%) 4.11s (± 0.50%) ~ 4.08s 4.14s p=0.935 n=6
Bind Time 1.59s (± 0.62%) 1.59s (± 0.62%) ~ 1.58s 1.61s p=0.933 n=6
Check Time 22.34s (± 0.31%) 22.52s (± 0.38%) +0.18s (+ 0.79%) 22.38s 22.62s p=0.016 n=6
Emit Time 1.72s (± 1.63%) 1.73s (± 1.34%) ~ 1.70s 1.77s p=0.462 n=6
Total Time 29.76s (± 0.37%) 29.95s (± 0.28%) +0.19s (+ 0.66%) 29.80s 30.01s p=0.013 n=6
vscode - node (v18.15.0, x64)
Memory used 2,908,000k (± 0.00%) 2,907,952k (± 0.00%) ~ 2,907,786k 2,908,081k p=0.471 n=6
Parse Time 12.93s (± 0.34%) 12.95s (± 0.35%) ~ 12.89s 13.01s p=0.571 n=6
Bind Time 4.15s (± 0.34%) 4.16s (± 0.33%) ~ 4.14s 4.18s p=0.677 n=6
Check Time 72.33s (± 0.50%) 72.48s (± 0.78%) ~ 71.60s 73.21s p=0.521 n=6
Emit Time 20.10s (± 8.14%) 21.38s (±10.21%) ~ 19.24s 23.68s p=0.688 n=6
Total Time 109.50s (± 1.70%) 110.95s (± 2.43%) ~ 107.90s 113.45s p=0.378 n=6
webpack - node (v18.15.0, x64)
Memory used 409,046k (± 0.01%) 409,055k (± 0.02%) ~ 408,908k 409,154k p=0.810 n=6
Parse Time 4.81s (± 0.74%) 4.80s (± 0.50%) ~ 4.77s 4.84s p=0.294 n=6
Bind Time 2.06s (± 0.87%) 2.06s (± 1.24%) ~ 2.02s 2.08s p=1.000 n=6
Check Time 20.96s (± 0.44%) 20.87s (± 0.33%) -0.09s (- 0.41%) 20.79s 21.00s p=0.037 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.84s (± 0.33%) 27.73s (± 0.23%) ~ 27.64s 27.80s p=0.092 n=6
xstate - node (v18.15.0, x64)
Memory used 513,675k (± 0.02%) 513,641k (± 0.02%) ~ 513,491k 513,746k p=0.471 n=6
Parse Time 4.90s (± 0.97%) 4.92s (± 0.88%) ~ 4.87s 4.98s p=0.423 n=6
Bind Time 2.29s (± 0.71%) 2.31s (± 1.32%) ~ 2.28s 2.36s p=0.332 n=6
Check Time 4.32s (± 0.52%) 4.30s (± 0.90%) ~ 4.26s 4.37s p=0.170 n=6
Emit Time 0.11s (± 0.00%) 0.11s (± 0.00%) ~ 0.11s 0.11s p=1.000 n=6
Total Time 11.62s (± 0.49%) 11.63s (± 0.56%) ~ 11.56s 11.70s p=0.748 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,560ms (± 0.36%) 3,523ms (± 0.60%) -37ms (- 1.05%) 3,485ms 3,544ms p=0.010 n=6
Req 2 - geterr 8,318ms (± 0.80%) 7,611ms (± 0.72%) 🟩-707ms (- 8.49%) 7,566ms 7,711ms p=0.005 n=6
Req 3 - references 475ms (± 0.96%) 431ms (± 0.50%) 🟩-44ms (- 9.24%) 428ms 433ms p=0.005 n=6
Req 4 - navto 407ms (± 0.22%) 341ms (± 0.78%) 🟩-66ms (-16.30%) 336ms 343ms p=0.005 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 131ms (± 0.79%) 114ms (± 1.47%) 🟩-17ms (-12.76%) 111ms 115ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,727ms (± 0.73%) 3,669ms (± 1.21%) ~ 3,620ms 3,737ms p=0.066 n=6
Req 2 - geterr 6,144ms (± 1.37%) 5,635ms (± 0.76%) 🟩-509ms (- 8.28%) 5,585ms 5,711ms p=0.005 n=6
Req 3 - references 492ms (± 0.63%) 443ms (± 0.67%) 🟩-49ms (- 9.90%) 440ms 448ms p=0.005 n=6
Req 4 - navto 445ms (± 0.95%) 341ms (± 3.17%) 🟩-104ms (-23.28%) 336ms 363ms p=0.005 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 110ms (± 3.03%) 102ms (± 0.82%) 🟩-8ms (- 7.45%) 100ms 102ms p=0.004 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,905ms (± 1.00%) 3,819ms (± 0.58%) -86ms (- 2.20%) 3,790ms 3,850ms p=0.005 n=6
Req 2 - geterr 2,465ms (± 1.44%) 2,268ms (± 0.48%) 🟩-197ms (- 7.99%) 2,251ms 2,282ms p=0.005 n=6
Req 3 - references 144ms (± 1.73%) 131ms (± 1.73%) 🟩-14ms (- 9.58%) 128ms 133ms p=0.005 n=6
Req 4 - navto 557ms (± 1.07%) 531ms (± 1.37%) 🟩-26ms (- 4.69%) 523ms 543ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 453ms (± 0.56%) 439ms (± 1.27%) 🟩-14ms (- 3.02%) 432ms 446ms p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 222.77ms (± 0.16%) 222.61ms (± 0.18%) -0.16ms (- 0.07%) 220.62ms 228.04ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.65ms (± 0.16%) 228.64ms (± 0.19%) ~ 227.01ms 239.29ms p=0.338 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 273.78ms (± 0.30%) 273.90ms (± 0.31%) ~ 266.35ms 283.77ms p=0.076 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 332.39ms (± 0.28%) 332.32ms (± 0.29%) ~ 323.85ms 337.56ms p=0.393 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Now that the benchmarker can test the public API as tsc:

image

@jakebailey jakebailey merged commit 3456c6e into microsoft:main Apr 5, 2024
25 checks passed
@jakebailey jakebailey deleted the mono-services branch April 5, 2024 00:03
@jakebailey jakebailey mentioned this pull request Apr 5, 2024
Draft
@rbuckton rbuckton mentioned this pull request Apr 6, 2024
Open
This was referenced Jul 2, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@weswigham weswigham weswigham left review comments

@rbuckton rbuckton rbuckton approved these changes

Assignees

@jakebailey jakebailey

Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jakebailey @typescript-bot @rbuckton @weswigham