Unable to use lmdb during documentation generation

[vEpiphyte]

Affected Operating Systems

• Linux <-- Confirmed

Affected py-lmdb Version

>>> print(lmdb.__version__)
0.93

py-lmdb Installation Method

pip install lmdb
installed into a virtualenv managed by pyenv

Using bundled or distribution-provided LMDB library?

unknown

Distribution name and LMDB library version

>>> print(lmdb.version())
(0, 9, 18)

Machine "free -m" output

N/A

Other important machine info

N/A

Describe Your Problem

When using a custom script to extend Sphinx documentation generation, I am unable to use lmdb during that process.

Errors/exceptions Encountered

Traceback (most recent call last):
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/sphinx/application.py", line 234, in __init__
    self._init_builder()
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/sphinx/application.py", line 312, in _init_builder
    self.emit('builder-inited')
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/sphinx/application.py", line 489, in emit
    return self.events.emit(event, self, *args)
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/sphinx/events.py", line 79, in emit
    results.append(callback(*args))
  File "conf.py", line 358, in run_autodoc
    s_autodoc.main(['--configable-opts', '--savefile', 'synapse/configables.rst'])
  File "../synapse/tools/autodoc.py", line 388, in main
    return docConfigables(outp, fd)
  File "../synapse/tools/autodoc.py", line 132, in docConfigables
    obj = cls(fp, conf=conf)  # type: s_config.Config
  File "../synapse/neuron.py", line 94, in __init__
    self.kvstor = s_kv.KvStor(self._path('cell.lmdb'))
  File "../synapse/lib/kv.py", line 253, in __init__
    self.lenv = lmdb.open(path, writemap=True, max_dbs=16)
  File "/home/someuser/.pyenv/versions/synapse_353_docs/lib/python3.5/site-packages/lmdb/cffi.py", line 686, in __init__
    envpp = _ffi.new('MDB_env **')
NameError: name '_ffi' is not defined

Describe What You Expected To Happen

I would have expected to be able to use the lmdb module during documentation generation.

Describe What Happened Instead

Unable to use lmdb during document generation.

User Diagnosis

During import, __intit__.py falls back to using the cffi lmdb implementation due to the _reading_docs() function. Later, in cffi.py, that is checked again and used to derming if the cffi binding is actually done. If we are being executed by pydoc or sphinx-build, the user cannot use lmdb.

Proposed Solution

There is currently a check for LMDB_FORCE_CPYTHON as a environmental variable but that doesn't seem to allow us to use that to force trying to load the cpython bound libraries. Allowing the user to provide that envar and using it to force the cpython bound libraries to be used during documentation generation would allow the use of the library someone using sphinx. If that is acceptable, I'd be happy to put together a PR which supports that.

Related Issues

This appears related to #15

[vEpiphyte]

For reference, in the Makefile created by sphinx, replacing sphinx-build with python -m sphinx does temporarily resolve this issue since _reading_docs() will return False.

[vEpiphyte]

The above hack doesn't work with readthedocs since they ignore your makefile :(

[scossu]

Same problem here. @vEpiphyte 's workaround works locally. Is there a way to make it work with readthedocs while a proper solution is implemented?

[vEpiphyte]

@scossu - the project i am working on uses a custom docs generation tool for certain classes. That is invoked by out conf.py. I ended up doing some hackery with subprocess to execute our autodoc tool using python directly. You can refer to the implementation here https://github.com/vertexproject/synapse/blob/v0.0.49/docs/conf.py#L359

[scossu]

Thanks @vEpiphyte . Any chance that the PR / solution you mention in your issue description be implemented?

[scossu]

@vEpiphyte Unfortunatly I am unable to discern the code specific to your framework from the code required to make document generation work. I have little experience with Sphinx. Would you be so kind to indicate the minimum setup to bypass the issue? Thanks a lot.

[scossu]

For the record, this seems to be a relatively portable workaround.

Before importing any other modules requiring LMDB in conf.py, add:

class MockModule(MagicMock):
    @classmethod
    def __getattr__(cls, name):
        return MagicMock()

MOCK_MODULES = ['lmdb']
sys.modules.update((mod_name, MockModule()) for mod_name in MOCK_MODULES)

[vEpiphyte]

@scossu I haven't spent any time towards tweaking the lmdb behavior. With that, it seems like you would not be able to actually use lmdb in any code that does document generation. is that correct?

[scossu]

@vEpiphyte I guess so; since this workaround is in the Sphinx config file, only doc generation should be affected. Pytest and such still import lmdb and work normally.

[scossu]

By the way, you suggest enabling the use of an environment variable to alter the import mechanism; would you be able to control that on readthedocs?

[vEpiphyte]

@scossu mock wouldn't work for me since i actually need to spin up lmdb environments during our autodoc process. its entirely possible the idea i initially had may not work with readthedocs - i had thought of that prior to actually having working solution in place.