Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MDB_BAD_RSLOT after fork due to spare txn #346

Open
callumwalker opened this issue Sep 15, 2023 · 0 comments · May be fixed by #363
Open

MDB_BAD_RSLOT after fork due to spare txn #346

callumwalker opened this issue Sep 15, 2023 · 0 comments · May be fixed by #363

Comments

@callumwalker
Copy link

Affected Operating Systems

  • Linux

Affected py-lmdb Version

1.4.0

py-lmdb Installation Method

sudo pip install lmdb==1.4.0

Also built from source to enable debugging.

Using bundled or distribution-provided LMDB library?

Bundled

Distribution name and LMDB library version

(0, 9, 29)

Describe Your Problem

Hi,

Using LMDB in a program that forks but which does not use LMBD inside the fork is giving me the following error

Traceback (most recent call last):
  File "/Servers/bcde/lmdb_test.py", line 25, in <module>
    txn = env.begin(write=False)
lmdb.BadRslotError: mdb_txn_renew: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

I've written the following testcase to reproduce the issue

import lmdb
import os
import time

print("      python: Opening environment")
env = lmdb.Environment('db.lmdb')
print("      python: Opened environment %r" % env)
print()

print("      python: Starting transaction")
txn = env.begin(write=False)
print("      python: Started transaction %r" % txn)
print()

print("      python: Deleting transaction")
del txn
print("      python: Deleted transaction")
print()

if (os.fork() != 0):
    os.wait()
    print("      python: Exited forked process")
    print()
    print("      python: Starting transaction (no. 2)")
    txn = env.begin(write=False)
    print("      python: Started transaction (no. 2) %r" % txn)
else:
    print("      python: Inside forked process")

On 1.4.0 this is giving me the following output

python: Opening environment
lmdb.cpython: env_new:1301: mdb_env_open(0x561033edf610, 'db.lmdb', 2097152, 644);
lmdb.cpython: db_from_name:981: DbObject '(null)' opened at 0x7f3b9fe5ce90
lmdb.cpython: env_new:1311: EnvObject 'db.lmdb' opened at 0x7f3b9fe4cc30
      python: Opened environment <Environment object at 0x7f3b9fe4cc30>

      python: Starting transaction
lmdb.cpython: make_trans:829: make_trans(env=0x7f3b9fe4cc30, parent=(nil), write=0, buffers=0)
      python: Started transaction <Transaction object at 0x7f3bae671ab0>

      python: Deleting transaction
lmdb.cpython: trans_dealloc:3242: 0x7f3bae671ab0: caching trans
lmdb.cpython: trans_dealloc:3248: 0x7f3bae671ab0: deleting trans
lmdb.cpython: trans_clear:3208: 0x7f3bae671ab0: clearing trans
lmdb.cpython: trans_clear:3218: 0x7f3bae671ab0: db is/was 0x7f3b9fe5ce90
      python: Deleted transaction

      python: Inside forked process
lmdb.cpython: env_clear:1118: 0x7f3b9fe4cc30: env_clear
lmdb.cpython: invalidate:340: invalidating parent=0x7f3b9fe4cc30 child 0x7f3b9fe5ce90
lmdb.cpython: env_clear:1126: 0x7f3b9fe4cc30: killing spare txn 0x561033f635b0
lmdb.cpython: txn_abort:3199: 0x561033f635b0: aborting
lmdb.cpython: env_clear:1132: Closing env
      python: Exited forked process

      python: Starting transaction (no. 2)
lmdb.cpython: make_trans:829: make_trans(env=0x7f3b9fe4cc30, parent=(nil), write=0, buffers=0)
lmdb.cpython: make_trans:859: using cached txn
Traceback (most recent call last):
  File "/Servers/bcde/lmdb_test.py", line 25, in <module>
    txn = env.begin(write=False)
lmdb.BadRslotError: mdb_txn_renew: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
lmdb.cpython: env_clear:1118: 0x7f3b9fe4cc30: env_clear
lmdb.cpython: invalidate:340: invalidating parent=0x7f3b9fe4cc30 child 0x7f3b9fe5ce90
lmdb.cpython: env_clear:1132: Closing env

If I comment out caching the transaction in trans_dealloc I get

      python: Opening environment
lmdb.cpython: env_new:1301: mdb_env_open(0x558d9790a610, 'db.lmdb', 2097152, 644);
lmdb.cpython: db_from_name:981: DbObject '(null)' opened at 0x7fc25e690da0
lmdb.cpython: env_new:1311: EnvObject 'db.lmdb' opened at 0x7fc25e680c30
      python: Opened environment <Environment object at 0x7fc25e680c30>

      python: Starting transaction
lmdb.cpython: make_trans:829: make_trans(env=0x7fc25e680c30, parent=(nil), write=0, buffers=0)
      python: Started transaction <Transaction object at 0x7fc26cea5ab0>

      python: Deleting transaction
lmdb.cpython: trans_dealloc:3248: 0x7fc26cea5ab0: deleting trans
lmdb.cpython: trans_clear:3208: 0x7fc26cea5ab0: clearing trans
lmdb.cpython: txn_abort:3199: 0x558d9798e5b0: aborting
lmdb.cpython: trans_clear:3218: 0x7fc26cea5ab0: db is/was 0x7fc25e690da0
      python: Deleted transaction

      python: Inside forked process
lmdb.cpython: env_clear:1118: 0x7fc25e680c30: env_clear
lmdb.cpython: invalidate:340: invalidating parent=0x7fc25e680c30 child 0x7fc25e690da0
lmdb.cpython: env_clear:1132: Closing env
      python: Exited forked process

      python: Starting transaction (no. 2)
lmdb.cpython: make_trans:829: make_trans(env=0x7fc25e680c30, parent=(nil), write=0, buffers=0)
      python: Started transaction (no. 2) <Transaction object at 0x7fc26cea5ab0>
lmdb.cpython: trans_dealloc:3248: 0x7fc26cea5ab0: deleting trans
lmdb.cpython: trans_clear:3208: 0x7fc26cea5ab0: clearing trans
lmdb.cpython: txn_abort:3199: 0x558d97928300: aborting
lmdb.cpython: trans_clear:3218: 0x7fc26cea5ab0: db is/was 0x7fc25e690da0
lmdb.cpython: env_clear:1118: 0x7fc25e680c30: env_clear
lmdb.cpython: invalidate:340: invalidating parent=0x7fc25e680c30 child 0x7fc25e690da0
lmdb.cpython: env_clear:1132: Closing env

As I'm not actively using LMDB inside the forked process I believe that this should be something that LMDB supports but maybe I need to change my code to explicitly close the environment before forking.
callumwalker pushed a commit to callumwalker/py-lmdb that referenced this issue Jun 12, 2024
Fix jnwatson#346: MDB_BAD_RSLOT after fork due to spare txn
658b193
@callumwalker callumwalker linked a pull request Jun 21, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix transactions freed by deallocation in child process callumwalker/py-lmdb
1 participant
@callumwalker