9 awesome real world applications of Reinforcement Learning

Reinforcement Learning is a framework for sequential decision making. It differs from the usual supervised setting as there are no labels present. In this framework an ‘agent’ interacts with an ‘environment’ to gain experience from which the agent learns to perform the most optimal action which maximises its rewards.

Reinforcement Learning has had major advances in the last few years and is being applied to real world problems. The pace of adoption has definitely been slower than other ML approaches as RL has its own challenges. The main barriers of applying RL to the real world is online learning in a real system is often expensive and not feasible which means we have to fall back on simulation and historical data to build models. This approach is more complicated as simulators capturing real world dynamics are difficult to build and historical data is always limited but despite this limitation RL has found many successful applications. In this article we will look at 9 amazing applications of Reinforcement learning in the real world.

1. Nuclear Fusion

Nuclear Fusion is a reaction when multiple atomic nuclei are combined to form different atomic nuclei. This process releases massive amounts of energy e.g our Sun generates energy by nuclear fusion of hydrogen into helium. Fusion is ‘clean energy’ i.e it produces no toxic by products like fission. If successfully harnessed fusion has the potential to revolutionise our future as it is an almost endless source of clean energy.

Fusion requires the reaction to be confined in a space while the reaction takes place. Tokamak is a torus-shaped device which uses a powerful magnetic field to confine plasma in the shape of a torus. This is one of the several methods of magnetic confinement devices.

Magnetic control of tokamak plasmas through deep reinforcement learning

This paper talks about a RL-designed magnetic controller and experimentally verify its performance on a tokamak. DeepMind led this effort to model and apply the model on a real Tokamak. This is one of the most complex real-world system to which RL has been applied.

2. Designing Chips (ICs)

IC ( Integrated Circuits ) are the basis of today’s computer and electronic systems. Chips take enormous effort to design and optimise (as the number of transistors increases ). Moore’s Law has also been slowing down which means that we have to come up with better methods & ideas to design chips. One of the recent approaches is to use AI to design more efficient chips.

Two such applications of RL in this domain stand out and they are the following

Chip Placement with Deep Reinforcement Learning

This approach is by Google and they have used it to design their TPU accelerators for training AI models. The main idea here is to use RL to optimise placement of circuit elements on the chip to optimise for power and performance.

The model is able to generate optimised chip layouts in a few hours as compared to the human experts ( 100s of them ) who take several weeks to generate the placements. The model uses a combination of RL and historical datasets.

The blog and the corresponding github repo for this are here & here.

PrefixRL: Optimization of Parallel Prefix Circuits using Deep Reinforcement Learning

This approach is by Nvidia.

3. ChatGPT ( Language Models )

ChatGPT is most recent model by OpenAI which as taken the internet and the world by storm. Its a GPT3 type model with which you can interact in a chat type interface.

ChatGPT uses a method called RLHF ( Reinforcement Learning with Human Feedback ) to train the models.

There are open source github repositories which can allow us to train models like ChatGPT

• Transformer Reinforcement Learning (trl)
• RL4LMs

More details on the training process can be found in these blogs here & here.

4. Commercial Cooling Systems

This is a recent paper by DeepMind building on their earlier application of ML make Google’s datacenter more efficient. HVAC ( Heating , ventilation and air conditioning ) systems are responsible for significant portion of the global CO2 emissions and also consumes a large portion of the worlds electricity.

Controlling Commercial Cooling Systems Using Reinforcement Learning

Earlier papers like A Review of Deep Reinforcement Learning for Smart Building Energy Management talk about methods which apply these in simulation. The innovation in DeepMind’s approach is that they build on the earlier simulation approaches and use a combination of simulation and offline training to create a state of the art model. They not only create the model but also deploy the model on real building to see the practical impact of the model.

5. Inventory Management

Inventory management or control is at the heart of operations management. Here replenishment actions must be taken to minimise costs but despite years of research the solution to many inventory control problems is not known. Many of these problems are not analytically tractable hence solutions can’t be derived rather found computationally.

Deep Inventory Management

This is a work by Amazon where they apply RL to periodic review inventory control system. The paper discusses couple of novel ideas applied to this problem such as

• Use of historical data to create a simulator
• Use a novel algorithm called differentiable simulator.

The ideas presented in the paper are not just simulations or theoritical but has been deployed as mentioned in the paper and delivered significant gains in the inventory state.

6. Agriculture

At first sight it doesn’t seem like there are many ways RL can be applied to agriculture but if we look deeper we see that lot of agriculture falls into the domain of ‘sequential decision’ making. Every decision or action during agriculture has impact down the line for the farmer in terms of crop yield or output.

There are multiple decisions to be take while growing crop i.e amount of fertiliser to use and when, what amount of irrigation to have and when etc.. These decisions not enable one to optimise the crop yield but also optimise the use of fertiliser and water thus minimising the impact of our actions on the environment.

gym-DSSAT: a crop model turned into a Reinforcement Learning environment

DSSAT is a realistic crop growth simulator which takes in various parameters like fertiliser, irrigation and weather into consideration. DSSAT is used by farmers around the world for assistance in growing their crops. Gym-DSSAT is python wrapper around DSSAT which transform it into a gym environment. This environment can be then used to create RL agents which can give us optimal policies to grow crops.

7. Finance

Finance beautifully falls into the RL framework as many problems in the financial domain are sequential decision making in nature. There are many problems in finance where RL can be applied like

Portfolio Management

Portfolio Management is the process of selecting assets ( sometimes from different classes i.e stocks, bonds or cash etc.. ) to maximise a certain objective. This objective is usually to maximise profit along with some notion of risk management. Portfolio Management involves the following main steps

Key Paper : Detecting and adapting to crisis pattern with context based Deep Reinforcement Learning

Trading

Trading is buying and selling securities or financial instruments with the objective of earning a profit. Trading is also a sequential decision making problem and fits naturally in the framework of RL.

Key Paper : FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

8. Autonomous Driving

Autonomous driving is domain where RL has immense applications. RL is applied to the motion planning and control of autonomous vehicles.

Learning to Drive in a Day

This was one of the earliest papers to demonstrate how to apply RL end to end in autonomous driving. They demonstrated an agent which was able to learn how to drive from scratch using a randomly initialized neural network.

Using novel reward shaping the model was able to learn from a handful of episodes to drive and navigate successfully. This approach was actually tried on a live vehicle and demonstrated. The blog discussing the same is here and video of the demo is here.

9. Aviation

Aviation consists not only of commercial aviation but also military aviation and RL has a huge role to play here. There are multiple problems to which RL can be applied. Some of the key application areas are :

• Flight Planning and Scheduling
• Airline Maintenance
• AirTraffic Management
• Aircraft Control

The survey paper here talks about the above areas of applications and more.

Apart from the above applications in the commercial aviation sector some of the applications in the military domain are as follows

• Combat scenarios like dog fighting : Hierarchical Reinforcement Learning for Air-to-Air Combat & Harfang3D Dog-Fight Sandbox: A Reinforcement Learning research platform for the customized control tasks of fighter aircrafts.
• Control of Rocket Engines : A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines
• Missile Control : Missile guidance with assisted deep reinforcement learning for head-on interception of maneuvering target

References

• https://www.theverge.com/2022/12/13/23506086/nuclear-fusion-power-research-milestone-net-energy-gain
• https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver
• https://www.technologyreview.com/2022/05/27/1052826/ai-reinforcement-learning-self-driving-cars-autonomous-vehicles-wayve-waabi-cruise/
• https://medium.com/@mlblogging.k/rl-in-economics-794b43ffc995
• https://medium.com/@mlblogging.k/rl-in-finance-fcb8b2c31f9
• A Survey on Reinforcement Learning in Aviation Applications https://arxiv.org/pdf/2211.02147
• Deep Reinforcement Learning for Agriculture: Principles and Use Cases https://link.springer.com/chapter/10.1007/978-981-16-5847-1_4