Recursion error when multiprocessing.Pool.map bam file

[olgabot]

Describe the bug
Yet another bug to enjoy! This build is failing, but only for python 2.7.

To Reproduce
Steps to reproduce the behavior: Use multiprocessing (or pathos.multiprocessing, a lambda-friendly drop-in replacement for multiprocessing) to spawn multiple processes. Again, uses these bam files

import os

import bamnostic as bs
import pathos.multiprocessing as mp

folder = os.path.join('tests', 'test-data', '10x-example')

bam_file = bs.AlignmentFile(
        os.path.join(folder, 'possorted_genome_bam.bam'), mode='rb')

pool = mp.Pool(processes=2)
pool.map(lambda x: "CB" in x.tags, bam_file)

Expected behavior
Expected to iterate over the alignments in the bam file without error.

Screenshots

  File "/home/travis/build/dib-lab/sourmash/tests/sourmash_tst_utils.py", line 103, in runscript
    status = _runscript(scriptname)
  File "/home/travis/build/dib-lab/sourmash/tests/sourmash_tst_utils.py", line 50, in _runscript
    pkg_resources.load_entry_point("sourmash", 'console_scripts', scriptname)()
  File "/home/travis/build/dib-lab/sourmash/sourmash/__main__.py", line 77, in main
    cmd(sys.argv[2:])
  File "/home/travis/build/dib-lab/sourmash/sourmash/commands.py", line 278, in compute
    pool.map(lambda x: maybe_add_alignment(x, cell_seqs, args, barcodes), bam_file)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/multiprocess/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/multiprocess/pool.py", line 306, in map_async
    iterable = list(iterable)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 1420, in next
    read = bamnostic.AlignedSegment(self)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/core.py", line 199, in __init__
    bsize_buffer = self._io.read(4)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 832, in read
    self._load_block()  # will reset offsets
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 535, in _load_block
    self._buffers[self._block_start_offset] = self._buffer, block_size
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 516, in __setitem__
    self.cull()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 486, in cull
    self.popitem(last=False)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 178, in popitem
    value = self.pop(key)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 156, in pop
    result = self[key]
=== begin repeated many times ===
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 500, in __getitem__
    self[key] = value
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 516, in __setitem__
    self.cull()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 486, in cull
    self.popitem(last=False)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 178, in popitem
    value = self.pop(key)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 156, in pop
    result = self[key]
=== end repeated many times ===
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 498, in __getitem__
    if not key == list(self.keys())[-1]:
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 119, in keys
    return list(self)
RuntimeError: maximum recursion depth exceeded while calling a Python object

Desktop (please complete the following information):

• OS: macOS
• Python Version: 2.7
• bamnostic Version: v0.9.1

Additional context
None

[betteridiot]

Thank you again for making me aware of this. I had thought I fixed the recursion problem in 2.7, but it looks like I will have to take another look at it. Should have it fixed sometime tomorrow morning.

[betteridiot]

Okay...ugh, that was painful. v0.9.1rc3 should fix the recursion error on Python 2.7

I had to rewrite the way py27 OrderedDict works, and add the move_to_end method into it. There is also a minor change to bamnostic.bgzf to reflect the new api to bamnostic.utils.LruDict

[betteridiot]

Please keep up the good work

[betteridiot]

Nevermind...now my build is failing on tuple comparisons

[olgabot]

Haha I'm glad to help! Really appreciate the speed at which you're fixing these errors.

[betteridiot]

Finally done. v0.9.1rc4 has the newest release with recursion error corrections

[luizirber]

You're both doing a stellar job =]

[betteridiot]

BTW, thanks @luizirber for the recommendation you gave to @olgabot to look at bamnostic. She is really using the package and helping me find the holes in it.

[betteridiot]

v0.9.2 is out. This doesn't change anything of value except ensuring versioning matches that of PyPI and conda

[olgabot]

Great, thank you! --- Olga Botvinnik, PhD olgabotvinnik.com <http://www.olgabotvinnik.com>

…

On Tue, Sep 18, 2018 at 12:57 PM Marcus D Sherman ***@***.***> wrote: v0.9.2 <https://github.com/betteridiot/bamnostic/releases/tag/v0.9.2> is out. This doesn't change anything of value except ensuring versioning matches that of PyPI and conda — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxNcAk5Ktq7Kg2-4ZnZsKYnn4YWeTdtks5ucVAXgaJpZM4Ws4af> .