Performance issues

[karajan9]

I recently became interested in reinforcement learning, so I tried my luck with these environments by OpenAI. I noticed, however, quite a huge drop in performance in comparison to a Python version. On my computer 1000 runs of a random cartpole took about 16s in Julia, Python took about 4.8s for 10000 runs, which makes a factor of about 30x. With this kind of a difference it's woefully difficult to rely on Julia for these kind of simulations.

I am not very experienced in Julia; I know that calling Python from Julia isn't the fastest thing around, but are those factors usual or might I have a different problem going on?

If you need any more information or data, I'll happily provide both.

[JobJob]

Interesting. Can you post a gist of your code (in python and julia)?

[karajan9]

Thanks for coming back to me so quickly! Here is my (stripped down) Julia code:

using OpenAIGym

env = GymEnv("CartPole-v0")

function gen_data(N::Int)
    for i in 1:N
        reset!(env)
        for sars in Episode(env, RandomPolicy())
            state, action, reward, _ = sars
        end
    end
end

@time gen_data(1000)

This gave me the output for N = 1000:
15.943330 seconds (4.18 M allocations: 159.613 MiB, 0.50% gc time)
The code is adapted from https://github.com/CarloLucibello/DeepRLexamples.jl/blob/master/examples/reinforce_cartpole.jl

import gym

env = gym.make("CartPole-v0")

def gen_data(N):
    for i in range(N):
        env.reset()
        while True:
            action = env.action_space.sample()
            state, reward, done, info = env.step(action)
            if done == True:
                break

%time gen_data(10000)

Output for N = 10000:
CPU times: user 3.93 s, sys: 0 ns, total: 3.93 s
I modeled this code to be mostly the same as the Julia code.

I profiled the Julia code an most of the time it seemed to be spending in
PyCall/src/conversions.jl:808
but that doesn't mean too much to me, beyond what it says on the front. Maybe an issue with Python returning Lists instead of numpy Arrays (which I guess would make it faster)?

[JobJob]

Thanks for the minimal examples. I'd also noticed performance problems before but wasn't certain they were in this repo's code or in my code that was using it. Didn't get a chance to look into it at the time.

Anyway, did a little digging, think I've tracked down most of the issues. With some patches to this repo and PyCall, I have CartPole-v0 within 2x, and Pong-v4 running within 1.25x of the python speed.

Will probably only get around to making PRs to the respective repos next week though.

Cheers.

[JobJob]

I now have gym running as fast in Julia as in Python. PRs forthcoming here once JuliaPy/PyCall.jl#487 and a version of JuliaPy/PyCall.jl#486 get merged.

[JobJob]

@karajan9 see #13