15 awesome reinforcement learning environments you must know

1. Neural MMO

Neural MMO was initially released by OpenAI as an environment for testing multi agent reinforcement learning methods. MMO stands for Massively Multiplayer Online as this project is inspired by MMORPGs. The environment offers a optimised implementation where you can run hundreds of agents using just a CPU.

Documentation & tutorial are found here.

2. Android Env

Android Env was released by DeepMind. It’s an environment which provides a platform for building agents which interact with the android operating system. The library will return the pixel observations and rewards based on the tasks defined.

Using this environment agents can be trained to interact with the Android Env using RL and achieve tasks.

Setting up the environment docs and tutorials are found here.

3. MuJoCo

MuJoCo stands for Multi-Joint dynamics with Contact. It is released by DeepMind. It is general purpose physics engine that aims to facilitate research and development in robotics, biomechanics, graphics and animation, machine learning, and other areas which demand fast and accurate simulation of articulated structures interacting with their environment.

Documentation & tutorials are found here.

4. FinRL

FinRL is one of the first open source RL environments for finance. It can be integrated into various data sources either data feeds or user imported datasets or even simulated data. It supports various inbuilt RL algorithms which can be used

Tutorial are found here.

5. RecSim

RecSim is configurable system for applying Reinforcement learning to recommendor systems. RecSim simulates a recommender agent’s interaction with an environment consisting of a user model, a document model and a user choice model.

Documentation and tutorials are found here.

6. RLCard

RLCard is RL environment supporting various card games. It is developed by DATA Lab at Rice and Texas A&M University, and community contributors. It supports multiple games and algorithms like the following

Some of the games supported :

• BlackJack
• NoLimit Texas Hold’em
• Gin Rummy
• Bridge
• Dou Dizhu
• Uno

Some of Algorithms implemented are

• Counterfactual Regret minimization
• Neural Fictitious Self-Play
• Deep Q-Learning
• Deep Monte-Carlo

Documentation and tutorials are found here.

7. OpenSpiel

OpenSpiel is multi agent RL environment by DeepMind. OpenSpiel supports n-player (single- and multi- agent) zero-sum, cooperative and general-sum, one-shot and sequential, strictly turn-taking and simultaneous-move, perfect and imperfect information games, as well as traditional multi- agent environments such as (partially- and fully- observable) grid worlds and social dilemmas.

OpenSpiel supports multiple algorithms and games. Documentation and tutorials are found here.

8. AI Economist

AI Economist is an interesting research project by Salesforce looking into applying reinforcement learning for economics and socio-economic situations. The idea is to use this framework to come up with optimal economic policies. There have been few interesting papers from Salesforce on this topic.

Documentation & tutorials can be found here.

9. clubs_gym

Clubs_gym is a gym wrapper around the clubs python poker library. clubs is used for running arbitrary configurations of community card poker games. This includes anything from simple Leduc or Kuhn poker to full n-player No Limit Texas Hold’em or Pot Limit Omaha. This is a multi agent environment.

Documentation is found here.

10. Deep RTS

Deep RTS is simplified RL environment for real-time strategy games. The aim is to make RTS AI research more affordable by reducing the computation time of the environments. The environment is implemented in C++ and can run millions of steps per second

Deep RTS has inbuilt game scenarios which can be used for training the agents on specific tasks.

Examples on using Deep RTS can be found here.

11. Micro RTS

Gym-μRTS is similar to Deep RTS in objective. Existing full games have high computational costs which usually translates to thousands of hours of CPU & GPU time. This is a huge constraint for research.

Examples can be found here.

12. Nocturne

Nocturne is a RL environment released by Facebook Research to simulate multi agent systems. Nocturne is 2D driving simulation environment with partially observability. It’s built in C++ for speed. Currently Nocturne can handle data from Waymo Open Dataset but it can be modified to handle other data sources.

The image below shows a simulation of an intersection and also how an agent observes the environment. The agent has partial observability as its view is blocked by other vehicles.

Examples of how to use Nocturne can be found here.

13. CityFlow

CityFlow is a multi-agent reinforcement learning environment for large-scale city traffic scenario.

It has the following features

• A microscopic traffic simulator which simulates the behavior of each vehicle, providing highest level detail of traffic evolution.
• Supports flexible definitions for road network and traffic flow
• Provides friendly python interface for reinforcement learning
• Fast! Elaborately designed data structure and simulation algorithm with multithreading. Capable of simulating city-wide traffic. See the performance comparison with SUMO

Documentation and basic tutorials can be found here.

14. CyberBattleSim

CyberBattleSim is a framework for building abstract simulations of complex computer networks and systems in way such that reinforcement learning can be applied. It’s helpful to understand how agents interact and evolve in this complex environments. It is essentially gamification of network security such that reinforcement learning can be applied.

Documentation can be found here.

15. gym-dssat

Gym-dssat is RL environment for agricultural simulations. It is built upon the existing Decision Support System for Agrotechnology Transfer (DSSAT) and convereted into a python OpenAI Gym which allows the agent to interact with the crop at every stage of its growth.

The environment has three modes of operation

• Nitrogen mode for controlling fertiliser input
• Irrigation mode for controlling irrigation
• Third model combining nitrogen & irrigation mode

Apart from this the environment also supports a weather generator which can be randomly initialized to simulated different weather patterns. Tutorials on usage can be found here.

References

• https://www.microsoft.com/security/blog/2021/04/08/gamifying-machine-learning-for-stronger-security-and-ai-models/
• https://www.deepmind.com/publications/androidenv-the-android-learning-environment
• Practical deep reinforcement learning approach for stock trading https://arxiv.org/abs/1811.07522
• https://www.youtube.com/watch?v=8NCPqtPwlFQ Motivation, Core API, Brief Intro to Replictor Dynamics and Imperfect Information Games
• gym-DSSAT: a crop model turned into a Reinforcement Learning environment https://arxiv.org/abs/2207.03270
• https://github.com/Limmen/awesome-rl-for-cybersecurity
• https://ai.googleblog.com/2019/11/recsim-configurable-simulation-platform.html