Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Process-safe launching of minecraft instance to solve creating multiple minerl environments in parallel #352

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Process-safe launching of minecraft instance to solve creating multiple minerl environments in parallel #352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4f1eac2
launch instance lock to prevent build errors caused by multiple proce…
juliusfrost Jul 13, 2020
ddcf959
Merge branch 'master' into thread-safe-launch-instance
Jul 13, 2020
65c3e7a
use file lock instead of thread lock to become inter process safe
juliusfrost Jul 14, 2020
8dfd850
add filelock to requirements.txt
juliusfrost Jul 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion minerl/env/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import gym.envs.registration
import gym.spaces
import numpy as np
from filelock import FileLock
from lxml import etree
from minerl.env import comms
from minerl.env.comms import retry
Expand All @@ -47,6 +48,8 @@

missions_dir = os.path.join(os.path.dirname(__file__), 'missions')

launch_instance_lock = FileLock(os.path.join(os.path.dirname(__file__), 'env.lock'))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FileLock uses a file as communication for processes to proceed in the code. When a process enters the launch instance code, FileLock creates a file env.lock in the same directory and prevents other processes from entering unless the lock is removed.



class EnvException(Exception):
"""A special exception thrown in the creation of an environment's Malmo mission XML.
Expand Down Expand Up @@ -156,7 +159,8 @@ def _get_new_instance(self, port=None, instance_id=None):
if InstanceManager.is_remote():
launch_queue_logger_thread(instance, self.is_closed)

instance.launch()
with launch_instance_lock:
instance.launch()
return instance

def init(self):
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ getch>=1.0; sys_platform != 'win32' and sys_platform != 'cygwin'
coloredlogs>=10.0
matplotlib==3.0.3
dill>=0.3.1.1
filelock