Specific Plans for Replacing MuJoCo Environments With PyBullet

[jkterry1]

This is specifically for the MuJoCo class environments, not the robotic class environments that depend on MuJoCo (those will be replaced too, but that's a future problem).

The plan is to create a phys3d class for the pybullet replacement environments (since they aren't MuJoCo) with drop in replacements. Initially the environments will be unversioned and will have a warning message saying that they're beta. Once they've been thoroughly tested by the community to make sure that they're truly drop in replacements we'll add versioning to them and remove the MuJoCo class of environments.

For specific environments, the major source is is https://github.com/benelot/pybullet-gym. However, this repo has problems:
-It doesn't have the Humanoid Standup or Swimmer environments
-It says that the Walker2d and HalfCheetah environments are not similar to the reference implementations

This brings us to the lesser used source:
https://github.com/benelot/bullet-gym (which by the same guy, but is unmaintained and according to the environment description is the experimental version of the above repo). This is notable in that it has the missing Humanoid Standup and Swimmer environments.

The plan accordingly is for @Rohan138 to add the pybullet environments that replace MuJoCo ones to a new phys3d class from the first repo (it has many more than are included in Gym and I don't want to include new environments) and swimmer and humanoid from the second repo. Once these are in, we would greatly appreciate PRs that make the environments closer to drop in replacements for those in Gym and contributors testing these thoroughly.

[jkterry1]

@benelot if you'd be willing to help with this process, specifically regarding bugs that will inevitable arise after the beta is released and making the environments as similar to the MuJuCo ones as possible, I would greatly appreciate it.

[jkterry1]

So it turns out that Pybullet natively includes some of these environments:

https://docs.google.com/document/d/10sXEhzFRSnvFcl3XxNGhnD4N2SedqwdAvK3dsihxVUA/edit# (page 67). I have no idea if they wrote them from scratch or if they took them another source, or if they're more maintained than any of those other sources. Based on the included environments I'm assuming they're from the first repo. I would rather these live in Gym, I'll create an issue with them.

[jkterry1]

So apparently the PyBullet repo (https://github.com/bulletphysics/bullet3) has disabled issues and only allows for PRs, which they don't generally respond to? @Rohan138 figure out if we should be stealing code from them or from the other sources and then create a PR there to remove the environments we're including from bullet master.

[benelot]

I can certainly provide clarification with some of the issues here: I started a rewrite of the MuJoCo envs based on pybullet to make them base on an open source physics framework (open science and stuff). Erwin Coumans from pybullet started providing support for some google internal envs, so I asked him if he would be willing to merge some of my code into his repository. I am not sure if they are still there of if my code is now the only source of truth.

My note on some envs not being similar to the original MuJoCo ones is related to the fact that I did not find a document explaining some of the mujoco observations, so I could not find corresponding observations in pybullet. If you can help me with that, we can certainly find correspondences.

I would certainly be willing to help with this process if I can, given that it has always been my goal to provide alternative open envs for the MuJoCo ones.

[jkterry1]

@benelot could you please email me at [email protected]?

[benelot]

@jkterry1 Sent! Did you get it?

[jkterry1]

yep :)

(for those watching I just wanted to set up a zoom meeting)

[Rohan138]

Just mentioning #2345 here; if we want drop-in replacements where the only change required is pip install gym[phys3d], then we may have to deal with collisions in environment ids. In general, I think this PR would benefit from the plugins system proposed.

[jkterry1]

Two things from the exchanges with @benelot needs to be mentioned for those watching because I don't believe it's been publicly documented:

1. The environments from repo python3 #1 that were upstreamed into pybullet have included a lot of pybullet maintenance, however the master repo python3 #1 has included various observation fixes after the merge, and these features have never been resolved. This means that results between repos python3 #1, fix typo in cloning installation #2 and the environments in pybullet should not be compared, and is a profound argument for upstreaming these and making these environments officially and highly reviewed. Also will have to figure out how to merge the code somehow, @Rohan138 I would prefer to move things from repo python3 #1 and fix typo in cloning installation #2 in first before adding the upgrades in pybullet (amongst other upgrades) so that those can review

2. The observation spaces of these are likely profoundly troubling, to qoute: "The most work intensive thing will be to identify the observations of Mujoco (what they are conceptually) and replace them with pybullet provided observations. The observations in my code contains a lot of zeros to fill in the blanks such that the code runs, but this certainly leads to slower or impossible training"

[sgillen]

Hello I also wanted to chime in. This is something I was briefly interested in awhile ago but didn't really go anywhere with.

I'm not quite sure what exactly you mean by a "drop in replacement" I think it will probably be possible to make the observation space in bullet mimic what the current MuJoCo environments, but I think it will be much more difficult to get the performance of RL algorithms across similar environments in the two different sims to be roughly equivalent.

Like I mentioned on the discord the contact / constraint model between the two simulators is very different, although we can tweak the parameters in bullet to try to better match what MuJoCo does. I made a small example doing this here: https://gist.github.com/sgillen/7674fe6b8254aa512f5091367719fef2 which may be interesting if replicating the MuJoCo envs as close as possible is necessary.

Another question I have is (and I mean this as a genuine question) why replicate the Mujoco environments exactly in the first place? Another option might be to try and upgrade / update the environments that @benelot developed / that live in pybullet_envs. I think those environments are also widely used at this point, but I have no numbers on that.

[sgillen]

Also, you may be aware of these already but I thought I'd bring them up just in case:

1. I think the bullet environments will be incompatible with the render() interface gym currently uses. Though maybe we can work around this ...

2. For the robotic environments (excluding shadow hand) this: https://github.com/qgallouedec/panda-gym repo seems pretty good, I know rl-baselines3-zoo already includes benchmarks for it alongside the mj versions, but I have not looked into them closely yet.

[vwxyzjn]

Related stuff… https://deepmind.com/blog/announcements/mujoco Maybe we can still keep MuJoCo after all.

[jkterry1]

Closed in favor of #2456