Optimize concurrent collection's shrink logic #3417
Conversation
|
Ping @eolivelli @lordcheng10 @hangc0276 PTAL, Thanks. |
|
rerun failure checks |
|
ping @eolivelli PTAL. |
|
ping @dlg99 PTAL. |
|
ping @shoothzj PTAL. |
| // so as to avoid frequent shrinking and expansion near initCapacity, | ||
| // frequent shrinking and expansion, | ||
| // additionally opened arrays will consume more memory and affect GC | ||
| int newCapacity = Math.max(alignToPowerOfTwo((int) (capacity / shrinkFactor)), initCapacity); |
There was a problem hiding this comment.
Why not just skip shrink when the size is smaller than the initCapacity? That would save a lot of computing of alignToPowerOfTwo((int) (capacity / shrinkFactor))
There was a problem hiding this comment.
Assuming that initCapacity is 256,mapFillFactor, mapIdleFactor, expandFactor, shrinkFactor keep the default values:
| initCapacity | capacity | mapFillFactor | mapIdleFactor | resizeThresholdUp | resizeThresholdBelow | expandFactor | shrinkFactor |
|---|---|---|---|---|---|---|---|
| 256 | 256 | 0.66f | 0.15f | 168 | 38 | 2 | 2 |
| 256 | 512 | 0.66f | 0.15f | 337 | 76 | 2 | 2 |
| 256 | 1024 | 0.66f | 0.15f | 675 | 153 | 2 | 2 |
| 256 | 2048 | 0.66f | 0.15f | 1351 | 307 | 2 | 2 |
When the capacity is expanded to 1024, the size needs to be less than 153 to shrink, and size=153 is less than initCapacity=256, causing at least 1024-153 of space wasted. This waste is worse when initCapacity is larger.
Therefore, we cannot use size to judge that it will not shrink when it is smaller than initCapacity.
@zymap PTAL
There was a problem hiding this comment.
This calculation will only happen if size < resizeThresholdBelow, it is worthwhile to use the occasional calculation to relieve the free space that is not being used.
There was a problem hiding this comment.
Thanks for your detailed explanation!
### Motivation Optimize concurrent collection's shrink and clear logic ### Changes 1. Reduce the repeated `Arrays.fill` in the clear process 2. When `capacity` is already equal to `initCapacity`,`rehash` should not be executed 3. Reduce the `rehash` logic in the `clear` process 4. Shrinking must at least ensure `initCapacity`, so as to avoid frequent shrinking and expansion near `initCapacity`, frequent shrinking and expansion, additionally opened `arrays` will consume more memory and affect GC. If this PR is accepted, I will optimize the same `concurrent collection's shrink and clear logic ` defined in pulsar. Related to #3061 and #3074 (cherry picked from commit a580547)
### Motivation Optimize concurrent collection's shrink and clear logic ### Changes 1. Reduce the repeated `Arrays.fill` in the clear process 2. When `capacity` is already equal to `initCapacity`,`rehash` should not be executed 3. Reduce the `rehash` logic in the `clear` process 4. Shrinking must at least ensure `initCapacity`, so as to avoid frequent shrinking and expansion near `initCapacity`, frequent shrinking and expansion, additionally opened `arrays` will consume more memory and affect GC. If this PR is accepted, I will optimize the same `concurrent collection's shrink and clear logic ` defined in pulsar. Related to apache#3061 and apache#3074
Motivation
Optimize concurrent collection's shrink and clear logic
Changes
Arrays.fillin the clear processcapacityis already equal toinitCapacity,rehashshould not be executedrehashlogic in theclearprocessinitCapacity, so as to avoid frequent shrinking and expansion nearinitCapacity, frequent shrinking and expansion, additionally openedarrayswill consume more memory and affect GC.If this PR is accepted, I will optimize the same
concurrent collection's shrink and clear logicdefined in pulsar.Related to #3061 and #3074