Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CRASH] dictRehash on eviction #876

Open
keithchew opened this issue Oct 13, 2024 · 0 comments
Open

[CRASH] dictRehash on eviction #876

keithchew opened this issue Oct 13, 2024 · 0 comments

Comments

@keithchew
Copy link

keithchew commented Oct 13, 2024

Testing on async_flash branch, but bug is likely present earlier. When memory is under pressure, a crash happens during the eviction process:

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
7:101:S 13 Oct 2024 04:33:16.151 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
7:101:S 13 Oct 2024 04:33:16.151 # Accessing address: 0x8065700082
7:101:S 13 Oct 2024 04:33:16.151 # Crashed running the instruction at: 0x55a96cc7b836

------ STACK TRACE ------
EIP:
/opt/KeyDB/bin/keydb-server *:6379(dictRehash+0x86) [0x55a96cc7b836]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f7de4eb2420]
/opt/KeyDB/bin/keydb-server *:6379(dictRehash+0x86) [0x55a96cc7b836]
/opt/KeyDB/bin/keydb-server *:6379(dictGetSomeKeys+0x81) [0x55a96cc7e261]
/opt/KeyDB/bin/keydb-server *:6379(evictionPoolPopulate(int, redisDb*, bool, evictionPoolEntry*)+0x9e) [0x55a96cd7d1fe]
/opt/KeyDB/bin/keydb-server *:6379(performEvictions(bool)+0xf3f) [0x55a96cd7f4ff]
/opt/KeyDB/bin/keydb-server *:6379(processCommand(client*, int)+0xf5c) [0x55a96cc8b8dc]
/opt/KeyDB/bin/keydb-server *:6379(processCommandAndResetClient(client*, int)+0x69) [0x55a96cca9cc9]
/opt/KeyDB/bin/keydb-server *:6379(processInputBuffer(client*, bool, int)+0x254) [0x55a96ccafe94]
/opt/KeyDB/bin/keydb-server *:6379(processClients()+0xe6) [0x55a96ccb0036]
/opt/KeyDB/bin/keydb-server *:6379(beforeSleep(aeEventLoop*)+0x2db) [0x55a96cc8701b]
/opt/KeyDB/bin/keydb-server *:6379(aeProcessEvents+0x400) [0x55a96cc78680]
/opt/KeyDB/bin/keydb-server *:6379(aeMain+0x3e) [0x55a96cc78e6e]
/opt/KeyDB/bin/keydb-server *:6379(workerThreadMain(void*)+0x12b) [0x55a96cc923db]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f7de4ea6609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f7de4dc9133]

Note that the server has this option:

activerehashing no

So, both code flows in active rehash and async rehash have been eliminated. I wonder if there could be a fundamental issue with the main dict logic.

Will try to narrow this down further, but thought I did post this first just in case there are others who have come across the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant