TF-Agents: A Flexible Reinforcement Learning Library for TensorFlow with example code implementation of Soft Actor-Critic

Reinforcement learning has become a trending topic among all the tech giants and none of them is sitting back to catch up on this. Till date, OpenAI with their large and versatile set of codes for the RL algorithms have been the most user-friendly whether it is switching between Gym environments or

transitioning between different RL algorithms.

https://github.com/openai/baselines

OpenAI Baselines is a set of high-quality implementations of reinforcement learning algorithms.

Source

These algorithms make it easier for the research community to replicate, refine, and identify new ideas to create good baselines to build research on top of. They have beautifully abstracted the details of the RL algorithms and have made the use of these techniques as easy as calling a single class and feeding it essential details like environment name and batch sizes. This has made experimentation much easier and implementation simpler for the people new to the field.

What is TF-Agents?

Tensorflow seeing such build-up, has started building up its repository for RL algorithms and then import them in the builds of tensorflow itself where it can be used via a single import of the library tf.agents. This will help a lot of people of directly using the code from the package itself and will probably become something like using different Neural Networks in the code.

Working of TF agents

In TF-Agents, the core elements of RL algorithms are implemented as Agents. An agent encompasses two main responsibilities: defining a Policy to interact with the Environment, and how to learn/train that Policy from collected experience.

Why use TF-Agents?

Reinforcement Learning Basics

1. Great for Learning and implementing RL algorithms
2. Jupyter notebooks have been given with example codes
3. They are well suited for solving complex problems which require a lot of tweaking
4. Have been tested well and work well with gin-config

Minitaur pybullet trained with TF Agents PPO

Installation:

To install the latest version, use nightly builds of TF-Agents under the pip package tf-agents-nightly

pip install --user --upgrade tf-agents-nightly

Run the above command to install tf-agents in your tensorflow package.

Code Implementation Example-

We will just go over the basics of how to use a Soft Actor-Critic from the TF-Agents library and then deploy it for any gym environment.

First, we import the following packages after having installed the tf-agents:

Next, we compose the agent for our SAC which is an actor-critic agent and so we will require two networks. The critic will give us value estimates for Q(s, a). That is, it will get as input observation and action, and it will give us an estimation of how beneficial that action was for the assigned state.

The Actor-Network will foretell parameters as a Normal distribution.

Now finally we can instantiate the agent with the two actor and critic networks-

Agents include two policies:

1. The agent policy
2. The behavioural policy that is used for data collection

For evaluation and deployment, we take the mean action by enclosing the main policy with Greedy Policy.

eval_policy = greedy_policy.GreedyPolicy(tf_agent.policy)
collect_policy = tf_agent.collect_policy

Finally, we have our agent ready and is ready to be deployed.

So in this blog post, I have introduced TF-Agents which is a versatile Reinforcement Learning Library for TensorFlow. Reinforcement learning is a type of machine learning meant to train software or agents to complete a task using positive and negative reinforcement. This training is done in real-time with continuous feedback to maximize the possibility of being rewarded and by using TF-Agents, you can have an amazing adaptability at the tip of your hands.

Below I’m mentioning few tutorials & references that might help you kickstart your TF-Agent journey !

Tutorials:

Tensorflow has provided some even more in-depth codes for Tf Agents on jupyter notebooks on various topics like-

1. Into to Reinforcement Learning
2. DQN tutorial
3. Using Environments in RL
4. What are Policies and their implementation
5. Driver required to import agents and packages
6. Using Replay buffer in RL
7. Reinforce Tutorial
8. Soft Actor-Critic on Minitaur

References

TF-Agents: A Flexible Reinforcement Learning Library for TensorFlow (Google I/O’19)

Also, refer to the TF agents repository for more details.