improve performance of HAMTs #4889
Conversation
ghost
assigned
unixfs/hamt/bitfield_test.go
Outdated
| t.Fatal("expected 1 bit set") | ||
| } | ||
|
|
||
| bint := new(big.Int).SetBytes(bf.Bytes()) |
There was a problem hiding this comment.
Make sure we get the same thing as before.
unixfs/hamt/bitfield.go
Outdated
| // Before returns the number of bits set *before* this bit. | ||
| func (bf bitfield) Before(i int) int { | ||
| idx, off := bf.offset(i) | ||
| cnt := bits.OnesCount8(bf[idx] & (0xFF >> (8 - off))) |
There was a problem hiding this comment.
So, technically, we could make this faster with some unsafe magic to walk over 8 bytes at a time (casting them to a single uint64). However, I'm not sure if that'll help all that much (and this is going to be faster than anything we do with bigints).
There was a problem hiding this comment.
SGWM but I would feel much better if @whyrusleeping looked at it too.
Stebalien
force-pushed
the
feat/faster-hamt
branch
2 times, most recently
from
3e05b77
to
30e42e9
Compare
License: MIT Signed-off-by: Steven Allen <[email protected]>
License: MIT Signed-off-by: Steven Allen <[email protected]>
1. Use a custom bitfield type instead of bigints. 2. Make iterating over a hamt *significantly* faster. License: MIT Signed-off-by: Steven Allen <[email protected]>
License: MIT Signed-off-by: Steven Allen <[email protected]>
Stebalien
force-pushed
the
feat/faster-hamt
branch
from
30e42e9
to
1907e66
Compare
|
I've split the bitfield out into a separate package. |
|
TIL: go's big int implementation's |
|
@Stebalien I wonder if you would get a performance improvement by using wordsize uint's instead of bytes, like the big int lib does. |
Yeah, I probably should. Originally, I was worried about the cost of converting to/from bytes and planned on doing this with an unsafe cast but that's probably unnecessary. |
So, it doesn't help with indexing but it does help significantly with counting bits. Unfortunately, it makes the conversion code messy as hell. I seel if I can clean it up a bit. |
|
@Stebalien are you switching it to 64bit or are we leaving it as it is? |
|
I've written several patches to do so but they stink. I'd rather merge this and fix that later (as this patch has better performance regardless). |
According to the go benchmarks (not testing the datastore, etc.), these changes
make iterating over HAMTs ~50x faster and creating them ~2x faster.
based on #4884, merge after.