Evaluate and Upgrade to Ml-Agents 3.0.0

[alhasacademy96]

We are considering upgrading our Unity project from ML-Agents 0.3.0 to 3.0.0. This new version introduces several significant changes, including namespace and API modifications, feature additions and removals, performance improvements, and new dependencies. This change could go hand in hand with the code modifications to use Gymnasium instead of Gym for new versions of AAI for training. However, at this point, little work has been done on this issue so far.

Changes in ML-Agents 1.0.0 in a nutshell are:

Supported Python versions

• Python 3.10.x is now the minimum.

Unity Editor Version (for backend csharp)

• Unity Editor 2023.x.x is now the minimum editor to work on Ml-Agents (we are currenly using 2021)

Namespace and API Changes

• Namespace Renaming: MLAgents to Unity.MLAgents
• API Modifications: Methods in the Agent class (e.g., Done() to EndEpisode()) have been deprecated and replaced (a big one as it would require a deep look into how the agent class operates now, and make the changes thereafter)

Feature Additions and Removals

• ONNX Export Support: Introduced for broader compatibility.
• Removal of Multi-GPU Training: May affect performance and training times for large-scale scenarios.

Performance and Usability Improvements

• Singleton Academy: Simplifies usage in complex scenes (this is a big one as it significantly makes it easier for me to use one instance of Academy in my code)
• Improved Observation Handling: More efficient memory usage and potentially faster training times (always a big plus but we'd test out how much of a difference it would actually make during testing on our end)

Compatibility and Upgrade Concerns

• Barracuda Dependency Update: Requires compatibility checks with Unity versions.
• Self-Play Mechanism: New feature for adversarial training scenarios (could be useful for training)

Implications for AAI

Code Refactoring

• Significant refactoring could be needed for compatibility with 1.0.0.
• Updating namespaces, method calls, and adapting to new APIs, where needed

Training Process Adjustments

• Addressing the removal of multi-GPU support (i'm not sure about this as never worked on it)
• Adapting to new training mechanisms like self-play (a big one)
• Adjusting hyperparameters as necessary (a few changes here and there will probably be needed on unity and python)

Model Compatibility

• Impact of ONNX export support and Barracuda updates.
• Potential need for retraining or converting models.

Performance Impact

• Analyzing the effect of optimizations and observation handling changes (during training)

Next Steps (carried out potentially in 2024)

1. Conduct a detailed analysis of the changes in 1.0.0.
2. Develop a migration strategy and outline what needs to be done on python and unity bases.
3. Test the new version in a controlled environment, both manual and training on new and old config files.
4. Perform more testing to iron out potential issues low-level.
5. Update documentation and training materials as needed.
6. Update dependencies if needed
7. Optional: check if Gym to Gymnasium is a possible easy modification on ml-agents 1.0.0.

(Let me know if i missed anything)

[github-actions]

This issue is stale because it has been open for 45 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 50 days since being marked as stale. Please open a new issue for related bugs.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.