Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Game freezing / general robustness to wrapped game engine #161

Open
vzhong opened this issue Apr 24, 2019 · 8 comments
Open

Game freezing / general robustness to wrapped game engine #161

vzhong opened this issue Apr 24, 2019 · 8 comments
Labels
bug Something isn't working

Comments

@vzhong
Copy link

vzhong commented Apr 24, 2019

I've noticed that when randomly sampling actions in real text games (e.g. ZMachine games not generated by TextWorld), the game inevitably freezes/segfaults at some point (It seems like Jericho has the same issue). Since TextWorld is a RL environment for text games, can we make it more robust to these problems? For example, having a timeout failure or game crashed failure?

Here's a code snippet that consistently freezes on my machine:

# !mkdir -p games
# !wget -q http://www.ifarchive.org/if-archive/games/zcode/Balances.z5 -O games/Balances.z5

import textworld
import itertools
import tqdm
import numpy as np

np.random.seed(0)
episodes = 50
episode_len = 500

scores = []

for episode in range(episodes):
    print('episode {}'.format(episode))
    env = textworld.start('./games/Balances.z5')
    env.reset()
    env.seed(0)
        
    verbs = [w.word for w in env._jericho.get_dictionary() if w.is_verb]
    nouns = [w.word for w in env._jericho.get_dictionary() if w.is_noun]
    actions = [' '.join(tup) for tup in list(itertools.product(verbs, nouns))]
    
    for step in tqdm.trange(episode_len):
        act = np.random.choice(actions)
        print(step, act)
        game_state, score, done = env.step(act)
        if done:
            break
    scores.append(score)
    
print(scores)
@MarcCote
Copy link
Contributor

Thank you for reporting this issue. I'm not surprised Jericho would have the same issue since TextWorld uses Jericho under the hood. If I remember correctly, I think @mhauskn is aware of that issue but I don't know if we had a solution for it?

@MarcCote MarcCote added the bug Something isn't working label Apr 25, 2019
@mhauskn
Copy link
Contributor

mhauskn commented Apr 25, 2019

Here's an attempt to recreate the issue in Jericho alone:

from jericho import *
import itertools
import tqdm
import numpy as np

np.random.seed(0)
episodes = 50
episode_len = 500
ROM='/home/mahauskn/workspace/text-agents/roms/balances.z5'

scores = []

for episode in range(episodes):
    print('episode {}'.format(episode))
    env = FrotzEnv(ROM, 0)
    env.reset()

    verbs = [w.word for w in env.get_dictionary() if w.is_verb]
    nouns = [w.word for w in env.get_dictionary() if w.is_noun]
    actions = [' '.join(tup) for tup in list(itertools.product(verbs, nouns))]

    for step in tqdm.trange(episode_len):
        act = np.random.choice(actions)
        print(step, act)
        game_state, score, done, _ = env.step(act)
        if done:
            break
    scores.append(score)

print(scores)

Running this script I observe a Fatal error:

episode 0
  0%|                                                                                                                                                                                                                                                              | 0/500 [00:00<?, ?it/s]0 burn chasm
1 taste them
2 swim valley
3 turn floor
4 leave scrolls
5 put buck-too
6 ride barker
7 swallow helistars
8 l seven
9 unlock gold
10 fondle winged
11 full x
12 wait gorse
13 sniff every
14 flip square
15 hear sixteen
16 ride ne
17 insert lighted
18 switch twelve
19 x s
20 murder nw
21 shed northeast
  4%|██████████▋                                                                                                                                                                                                                                         | 22/500 [00:00<00:02, 214.07it/s]22 hear both
23 pick carpet
24 smash carpet
25 cut eastepisode 0
  0%|                                                                                                                                                                                                                                                              | 0/500 [00:00<?, ?it/s]0 burn chasm
1 taste them
2 swim valley
3 turn floor
4 leave scrolls
5 put buck-too
6 ride barker
7 swallow helistars
8 l seven
9 unlock gold
10 fondle winged
11 full x
12 wait gorse
13 sniff every
14 flip square
15 hear sixteen
16 ride ne
17 insert lighted
18 switch twelve
19 x s
20 murder nw
21 shed northeast
  4%|██████████▋                                                                                                                                                                                                                                         | 22/500 [00:00<00:02, 214.07it/s]22 hear both
23 pick carpet
24 smash carpet
25 cut east
26 slice grimoire
27 bother then
28 climb scroll
29 sing n
30 destroy chasm
31 i daffodils
32 blow pile
33 describe an
34 brief comma,
35 verbose shiny
36 no self
37 switch great
38 restore feather
39 say then
40 attach eight
41 wear floor
42 c,cast cedarwood
43 full door
44 noscript bazaar
45 go pans
  9%|██████████████████████▍                                                                                                                                                                                                                             | 46/500 [00:00<00:02, 220.78it/s]46 inside bozbar
47 straddle her

Fatal error: Illegal object
26 slice grimoire
27 bother then
28 climb scroll
29 sing n
30 destroy chasm
31 i daffodils
32 blow pile
33 describe an
34 brief comma,
35 verbose shiny
36 no self
37 switch great
38 restore feather
39 say then
40 attach eight
41 wear floor
42 c,cast cedarwood
43 full door
44 noscript bazaar
45 go pans
  9%|██████████████████████▍                                                                                                                                                                                                                             | 46/500 [00:00<00:02, 220.78it/s]46 inside bozbar
47 straddle her

Fatal error: Illegal object

Is this consistent with what you're seeing @vzhong?

@mhauskn
Copy link
Contributor

mhauskn commented Apr 25, 2019

I've verified that the same behavior is present in the Frotz (the base emulator used by Jericho) and have filed an issue there (https://gitlab.com/DavidGriffith/frotz/issues/111).

It's a valid point that RL environments shouldn't hang or produce fatal errors for what should be valid actions.

@vzhong
Copy link
Author

vzhong commented Apr 26, 2019

Hey @mhauskn , there are a variety of ways in which these failures can manifest. I haven't seen Fatal error: Illegal object from Jericho before, but I have seen segfaults.

I'm not even certain that the problem lies in Frotz. We are wrapping environments (textworld) around environments (Jericho) around environments (Frotz) around environments (ZMachine). There are probably fatal bugs in ZMachine (which is not written to support randomly behaving RL agents), and it's probably overly optimistic to hope to get those fixed. The problem for TextWorld users is that this kills the entire Python stack, which is not ideal for RL. It's probably useful to do something on the TextWorld end to handle these failure cases.

One simulator-agnostic way is to wrap the simulator (e.g. Jericho or Frotz) in a separate process, such that one can issue time-out-attached commands to the simulator. The simulator can fail/time out, but this does not affect TextWorld. When these events happen, TextWorld can just tell the user that the simulator died and the user can decide what they want to do (e.g. discard the trajectory, assign a negative reward for killing the simulator).

@mhauskn
Copy link
Contributor

mhauskn commented Apr 27, 2019

I've updated Jericho (now version 1.2.0) to avoid the Python stack crashing when a game encounters a fatal error. Instead the episode will be reported as over and the emulator status will be marked as halted.

This change should address the issue with Balances, but will not fix games that hang or segfault. Running your script across all supported Jericho games revealed 2 hangs and 3 fatal errors. The 3 fatal errors should be addressed by this update, but for now I'd advise avoiding the showverb command that causes hangs in the other 2 games. Please open issues with Jericho if you encounter other hangs or segfaults.

@vzhong
Copy link
Author

vzhong commented Apr 28, 2019

Thank you @mhauskn !! I'll take a look soon

@MarcCote
Copy link
Contributor

Thanks @mhauskn, now I'll need to think what's the best way to expose that information in TextWorld.

Regarding the hangs, what @vzhong proposes (wrapping a subprocess) is probably the safest way to do it but the drawback is the overhead over inter-multiprocess communications. If I remember correctly Jericho's main focus is speed.

That said, in TextWorld we are already using subprocesses that to handle communications with the git-glulx interpreter, so we could also do it for Jericho.

@mhauskn
Copy link
Contributor

mhauskn commented Apr 29, 2019

I'm avoiding implementing subprocess wrapping in Jericho for now. My hope is that segfaults + hangs can be fixed. If fixing segfaults/hangs turns out not to be feasible, then I may reconsider in the future. That said, I totally understand if TW wants to subprocess-wrap Jericho for added safety.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants