Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No OSX support for multithreading #375

Open
mklarqvist opened this issue Jul 28, 2020 · 2 comments
Open

No OSX support for multithreading #375

mklarqvist opened this issue Jul 28, 2020 · 2 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@mklarqvist
Copy link
Contributor

What
Our implementation makes use of Multiprocessing.queue and more specifically qsize that is not implemented in OSX.
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Queue.qsize. Attempting to run on OSX will result in a fatal error and then our manually spawned threads will never join resulting in an infinite wait.

Minimally reproducible code:

import ml4cvd
from ml4cvd.arguments import tensormap_lookup
from ml4cvd.tensor_generators import TensorGenerator
import glob

# files = glob.glob("/tf/google-ecg-rest-38k-tensors/2020-03-14/*.hd5")
files = glob.glob("2020-03-14/*.hd5")
batch_size = 100
tgen = TensorGenerator(
    batch_size = batch_size, 
    input_maps = [tensormap_lookup('ukb.ecg.ecg_rest')], 
    output_maps = [tensormap_lookup('ukb.demographics.age_0')], 
    num_workers = 8, 
    paths = files, 
    cache_size = 0
)
iterator = iter(tgen)

(internal code spawns workers if this is the first iteration)

>>> next(iterator)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/miniconda3/envs/ml4cvd/lib/python3.6/site-packages/ml4cvd/tensor_generators.py", line 164, in __next__
    if self.stats_q.qsize() == self.num_workers:
  File "/opt/miniconda3/envs/ml4cvd/lib/python3.6/multiprocessing/queues.py", line 117, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError
@mklarqvist mklarqvist added bug Something isn't working enhancement New feature or request labels Jul 28, 2020
@mklarqvist
Copy link
Contributor Author

This is partially related to issue #352

@mklarqvist
Copy link
Contributor Author

This will be addressed when migrating to tf.data.Dataset-based approaches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant