Skip to content
This repository has been archived by the owner on Jul 19, 2023. It is now read-only.

Parallelize row groups #843

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

simonswine
Copy link
Collaborator

@simonswine simonswine commented Jul 11, 2023

This retrieves rowGroups in parallel, I don't think this is the way to go.

We rather should look at switchting the querying order to align completely with the block order (seriesId, then timeNanos)

❯ benchstat before.txt after.txt
name                                                                                                             old time/op    new time/op    delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       284ms ± 6%     269ms ± 1%   -5.25%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       289ms ± 8%     283ms ± 7%     ~     (p=0.151 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       145ms ±15%     129ms ± 2%  -10.77%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12          123ms ± 6%     107ms ± 1%  -13.13%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12            116ms ± 0%     106ms ± 5%   -8.92%  (p=0.016 n=4+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12          106ms ± 4%      95ms ± 6%  -10.19%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12            106ms ± 5%      99ms ± 7%     ~     (p=0.056 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       128ms ± 2%     121ms ± 3%   -5.52%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       836ms ± 4%     810ms ± 0%   -3.05%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12     1.53s ± 1%     1.53s ± 1%     ~     (p=0.690 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12       1.47s ± 1%     1.48s ± 3%     ~     (p=0.841 n=5+5)

name                                                                                                             old alloc/op   new alloc/op   delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       492MB ± 0%     522MB ± 1%   +5.98%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       466MB ± 1%     495MB ± 1%   +6.14%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       157MB ± 0%     186MB ± 1%  +18.44%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12          113MB ± 2%     142MB ± 0%  +25.31%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12            122MB ± 1%     148MB ± 5%  +21.11%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12          101MB ± 2%     129MB ± 2%  +27.60%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12            108MB ± 1%     138MB ± 3%  +26.92%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       158MB ± 1%     189MB ± 0%  +19.35%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       785MB ± 0%     813MB ± 1%   +3.56%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12    1.56GB ± 0%    1.59GB ± 0%   +1.87%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12      1.58GB ± 0%    1.61GB ± 0%   +1.65%  (p=0.008 n=5+5)

name                                                                                                             old allocs/op  new allocs/op  delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       66.6k ± 0%     67.3k ± 0%   +1.07%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       80.7k ± 0%     81.3k ± 0%   +0.80%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       90.4k ± 0%     91.2k ± 0%   +0.86%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12           124k ± 0%      125k ± 0%   +0.56%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12             113k ± 0%      114k ± 0%   +0.66%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12           111k ± 0%      112k ± 0%   +0.70%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12             108k ± 0%      109k ± 0%   +0.81%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       69.5k ± 0%     70.4k ± 0%   +1.33%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       1.30M ± 0%     1.30M ± 0%   +0.06%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12     3.12M ± 0%     3.12M ± 0%   +0.03%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12       3.09M ± 0%     3.09M ± 0%   +0.03%  (p=0.008 n=5+5)

@simonswine simonswine changed the title 20230711 parallelize row groups Parallelize row groups Jul 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant