[RLlib] Add connect four self-play example with pettingzo

[elliottower]

Why are these changes needed?

I closed the other PR as there was an unsigned commit in it. #32492

I made this because I feel it's important to have examples using pettingzoo with selfplay. I can adapt it to another example if that is preferred, I figured it may be helpful to see the differences between it and the openspiel example it's based off. The main difference is that this game uses pygame rendering, and could be extended to take interactive user input, for example (Farama-Foundation/PettingZoo#905)

From old PR:

This PR mirrors functionality from self_play_with_open_spiel.py (PR), but uses the PettingZoo implementation of connect four, rather than OpenSpiel.

I used SuperSuit wrappers rather than writing my own wrapper as it is cleaner and less error prone versus writing my own wrapper, and is more flexible.

Related issue number

I already closed my own issue (link) related to this as I got the implementation working for myself, but I figure it will be a good resource for anyone who wants to use PettingZoo with RLlib in the future. The tutorials using RLlib from PettingZoo's wiki are currently marked as outdated and I couldn't find any resources, so this would be a helpful step forward.

Checks

This is my first PR so I wasn't sure if I needed to add documentation elsewhere. I couldn't find any for the example I was basing it off so I figure it is okay. I also could not find tests for other examples so I figured that is fine as well. I tried running the ci tests locally but wasn't able to figure them out.

As a side note, I've updated my branch to the latest master, and there seem to be linting ssues in other files which I don't want to change, so I've forced the commits using no-verify. I can change them as well but I figured it would be cleaner to only change this single file, and let the linting issues be handled by somebody else. This is the return when I run ci linting tests:

Skipped 1 files
All done! ✨ 🍰 ✨
1 file left unchanged.
Checking docstyle...
Reformatted changed files. Please review and stage the changes.
Files updated:

doc/source/ray-air/examples/analyze_tuning_results.ipynb
doc/source/ray-air/examples/batch_forecasting.ipynb
doc/source/ray-air/examples/serving_guide.ipynb
doc/source/ray-air/examples/torch_image_example.ipynb
doc/source/ray-air/examples/torch_incremental_learning.ipynb
doc/source/ray-contribute/docs.ipynb
doc/source/tune/examples/tune-xgboost.ipynb

Linting changes failed.
Please make sure 'ci/lint/format.sh' runs with no errors before pushing.
If you want to ignore this and push anyways, re-run with '--no-verify'.
error: failed to push some refs to 'github.com:elliottower/ray.git'

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[elliottower]

As I commented in the other PR (#33470), I don't know how to debug the errors I'm getting in these PR pipelines, so any help with that would be appreciated. Would really like to contribute this and potentially other RLlib pettingzoo examples, but if nobody is willing to help I can just put them in PettingZoo's repo instead.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!