Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: BlobWriteChannelV2 - same throughput less GC #2110

Merged
merged 3 commits into from
Jul 13, 2023
Merged

feat: BlobWriteChannelV2 - same throughput less GC #2110

merged 3 commits into from
Jul 13, 2023

Commits on Jul 13, 2023

  1. feat: overhaul of WriteChannel Storage.writer 

    Use stable buffer allocation with laziness.

Leverage new JsonResumableSession to provide more robustness and easier
separation of concerns compared to BlobWriteChannel

* rename blobWriteChannel.ser.properties to the correct blobReadChannel.ser.properties

### Runtime improvments

Throughput is on par with the existing v1 implementation, however GC
impact has been lightened with the new implementation.

Below is the summary of the GC improvement between v1 and v2.

These GC numbers were collected while uploading 4096 randomly sized
objects, from 128KiB..2GiB across 16 concurrent threads, using a default
chunkSize of 16MiB.

| metric                          | unit   |           v1 |           v2 | % decrease |
|---------------------------------|--------|-------------:|-------------:|-----------:|
| gc.alloc.rate                   | MB/sec |     2240.056 |     1457.731 |     34.924 |
| gc.alloc.rate.norm              | B/op   | 955796726217 | 638403730507 |     33.207 |
| gc.churn.G1_Eden_Space          | MB/sec |     1597.009 |     1454.304 |      8.936 |
| gc.churn.G1_Eden_Space.norm     | B/op   | 681418424320 | 636902965248 |      6.533 |
| gc.churn.G1_Old_Gen             | MB/sec |      691.877 |       11.316 |     98.364 |
| gc.churn.G1_Old_Gen.norm        | B/op   | 295213237398 |   4955944331 |     98.321 |
| gc.churn.G1_Survivor_Space      | MB/sec |        0.004 |        0.002 |     50.000 |
| gc.churn.G1_Survivor_Space.norm | B/op   |      1572864 |       786432 |     50.000 |
| gc.count                        | counts |         1670 |         1319 |     21.018 |
| gc.time                         | ms     |        15936 |         9527 |     40.217 |

Overall allocation rate is decreased, while Old_Gen use is almost entirely eliminated.

```
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Debian-1deb11u1, mixed
mode, sharing)

-Xms12g -Xmx12g
```

All other java parameters are defaults.
    @BenWhitehead
    BenWhitehead committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    49c880c View commit details
    Browse the repository at this point in the history

  2. chore: review cleanup

    @BenWhitehead
    BenWhitehead committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    5cc8745 View commit details
    Browse the repository at this point in the history

  3. chore: make content rewinding relative to it's offset in the object

    @BenWhitehead
    BenWhitehead committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    9f00950 View commit details
    Browse the repository at this point in the history