[Feature] Add env wrapper for Unity MLAgents

[kurtamohler]

Description

Adds UnityMLAgentsWrapper and UnityMLAgentsEnv environment wrapper classes to control MLAgents environments from TorchRL.

Motivation and Context

close #1110

• Someone has raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2469

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 15 New Failures, 6 Unrelated Failures

As of commit dd93644 with merge base f411f93 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  /home/runner/work/_temp/3580911f-abc3-4c3f-9c21-7ac5fb8078a3.sh: line 5: 2121 Bus error (core dumped) $RUN_BENCHMARK /tmp/tmp.OEbpWzdvzM
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Bus error
• Continuous Benchmark / CPU Pytest benchmark (gh)
  /home/runner/work/_temp/e53b45f3-b550-4561-bd18-e2eb3e5d8bb1.sh: line 3: 2148 Bus error (core dumped) python -m pytest --benchmark-json output.json
• Continuous Benchmark / GPU Pytest benchmark (gh)
  Bus error
• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 93ba7aca5af69e63e2db8356c02058f95bb42c91adbc0bb1561b06270676fd0a /exec failed with exit code 134
• Libs Tests on Linux / unittests-pettingzoo / linux-job (gh)
  RuntimeError: Command docker exec -t ccbad9eebb14ff7bb5c6780e403ad1b8e02b96ae447cb39ebe43425489a2e8eb /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t 7f14c4ca3a493632e3299edbfff0a6bf33115c6529757a542774da10b322315b /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t e78c3232103de2291a550da0165428e19c5a756d325676c62646c75ed1147f90 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t 749205a05e2b3c1659351c7f333074835a36b144a764c5c81a69f0a203437c22 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t 3a52c77864b22888a9cda77746a5f4bdffe02303fb915b0278a6243f7d657573 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu-oldget (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t fc59a90129ea206b71e93a9c763df51bd1c8179b3323e5204680476ac3347c48 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.11, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 70e10dfcd940b1f6064a823b47b3204e26df13c7ffb35644389dd7d2eb9128ad /exec failed with exit code 1
• Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 7a012d6275104ba99372f5a8c91380aa247c372fed4854b36ee44589a561c61e /exec failed with exit code 1
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c2c1e1456e3ca3f2e19313afc889b207eba40a20e7e64ef47f80ebc6e03d463a /exec failed with exit code 1
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  ##[error]fatal: couldn't find remote ref refs/pull/2469/merge

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cpu (gh) (trunk failure)
  ##[error]Unable to find any artifacts for the associated workflow
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda11_8 (gh) (trunk failure)
  ##[error]Unable to find any artifacts for the associated workflow
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_1 (gh) (trunk failure)
  ##[error]Unable to find any artifacts for the associated workflow
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_4 (gh) (trunk failure)
  ##[error]Unable to find any artifacts for the associated workflow
• Libs Tests on Linux / unittests-robohive (3.9, 12.1) / linux-job (gh) (trunk failure)
  test/test_libs.py::TestRoboHive::test_robohive[franka_slide_random-v3-True-True]
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  test/test_tensordictmodules.py::TestGRUModule::test_gru_vmap_complex_model

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

That looks amazing!
Can you share a bit about the current limitations (eg, are the specs expected to be fixed?) and some context about the tests - I see a couple of mocks and pytest marks

[vmoens]

what's this?

[kurtamohler]

When UnityMLAgentsEnv is not given a path to a file or the name of a registered env, it waits for the user to press the play button in the unity editor or run a mlagents-compatible binary. Since we can't easily do that in CI, I figured we can just skip the test by default. To run the test, the cmd line arg --unity_editor has to be given. We could remove this test, but I found it useful while working on this PR to have a test that connects to whichever environment I have open in the unity editor

[vmoens]

gotcha!

[kurtamohler]

Can you share a bit about the current limitations (eg, are the specs expected to be fixed?) and some context about the tests - I see a couple of mocks and pytest marks

Current limitations are:

• The specs are expected to be fixed throughout the run, so no new agents, behaviors, or observations can be added during the run.
• All agents are expected to request a decision in the first step so that they all get included when the specs are built.
• Because of the previous two points, the "Basic" and "DungeonEscape" environments from the example environments included with MLAgents do not work with our wrapper classes. But the ~15 other example envs do work.
• I haven't added the group_map argument to the wrappers.

For the tests:

• MLAgents comes with a few built-in environment binaries that can be automatically downloaded and run, so we have a test that uses those. For now, I only enabled testing two of those environments because each download is 180 MB. But thankfully they just get downloaded to /tmp, which is cleared on shutdown. We could add more of them to the list.
• The mock tests are modeled after the tests in the mlagents repo. I figured if that's how they test their environment class, we might as well do it that way too. Also, the registered envs are still an "experimental" feature, so I thought maybe we shouldn't only rely on those.