Switch from smhasher to mmh3

[dan-blanchard]

A number of our tests needed to have updated results after this change, so I'm not sure if this is correct or not. The main issue with this change is that for some reason smhasher treats all results from MurmurHash3 as unsigned big-endian integers, whereas mm3's 64 bit hash function returns a pair of little-endian signed integers. smhasher supported specifying unsigned seeds of any size, but the conversion to C long values would only take the first 32-bits worth of the number because it uses the I format specifier, which has no overflow checking. There's a pull request to switch mmh3 to this same method that has been open for a long time, but I don't think it will be merged (hajimes/mmh3#6), so I manually try to do the same overflow handling by dividing by 2^32. I believe the results still differ because of the endianness difference in how the two libraries treat their results, but I couldn't quite figure out if any of our code needs to be updated for that (or to handle mmh3 returning signed ints).

[kbourgoin]

What are we still using probably for and what motivated this change?

[conradlee]

My motivation for this change was to stop casterisk from using smhasher because it's not compatible with pypy (whereas mmh3 is compatible with pypy). That motivation is gone now because I'm not using pypy anymore.

…

On Thu, May 11, 2017 at 8:40 PM, Keith Bourgoin ***@***.***> wrote: What are we still using probably for and what motivated this change? — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOGJXkoL66X0aqbE8qbAMz7ByJOy3X1ks5r41YUgaJpZM4NO4Xx> .

[kbourgoin]

@dan-blanchard this is probably fine, but it isn't my area of expertise. This is definitely going to change the results of hashes, so will that have consequences elsewhere? If this operates in structures that only exist in memory, it'll be fine. But if this is going to load an old HLL from somewhere and start adding hashes, then we're going to have problems.

[dan-blanchard]

Surprisingly, hajimes/mmh3#6 got merged, so I'm going to tweak this to just require mmh3>=2.4 and not need the workarounds I was using before.

[dan-blanchard]

I've updated this to just workaround the endianness differences between how mmh3 and smhasher interface with the C code and I'm getting exactly the same results as before, so this is way better now, and I'm going to merge.

[kbourgoin]

That's great!