Robotic Arm Control using Tensorflow

This article articulates all the details for implementing the proposed model in the previous article. The entire implementation has been done on Tensorflow 1.13.0. All the Major parts of the Implementation have been explained comprehensively in the article.

The Buffer Memory

For storing the details of the actions taken by the actor in a particular state and the reward received after performing this action have to be stored in a buffer in order to use them for training. This is called ‘Experience Replay’. The details are stored in the memory in a tuple form : <s,a,s’,r>.
Here, s denotes the current state, a denotes the action take by the actor, s’ denotes the next state reached after performing the action and r denotes the reward received from the environment.

Actor Class

The implementation of Actor class has been shown below. A deep neural network is modeled that can take the state as an input and predicts the action from the given state. The state dimension and actor dimension should be passed as the parameters in the actor class. action_bound is used to clip the outputs of the actor network based on the

Critic Class
The implementation of Critic class has been shown described below. The critic class takes the input as a combination of state and action [state + action]. The deep neural network is constructed to predict the Q value for the given (state,action) pair. The expected reward is calculated and the updates in the critic network parameters are made after collecting the reward from environment. TD-error is calculated for the updation.

Some details about Robotic Arm Environment

Various platforms provide the simulation environment which is required for testing the developed Reinforcement Learning algorithm. Some the commonly used environments have mentioned below.

1. Mujoco : It is vastly used environment for creation of robotic arm models. It also supports the addtion of various Physics parameters. Also, the model can be exported to the outer environment and can be controlled with proper script.
2. PyBullet : It the Physics based environment specially supporting the Python language. It also supports creating of Robotic Arms however, the interfacing with other languages is a bit complex and hard.
3. Unity 3D : It is the most commonly used environment for the creation of any model. With addition ml-agents library, Unity seems to be the best environment for the ones who do not want to write an entire Model and train it manually. ml-agents provide support for Deep RL based algorithms and also provides the written scripts for them. The user just have to change the hyper parameter according the requirement and it is good to go.
4. Blender : Blender is used for creation of various 3D models in Mechanics. It has a built in support for the Python Language. This makes Blender the most suitable environment for the RL work. However, Blender lacks various functionality when it comes the computation part. Multi-threading is not explicitly supported in the blender. Moreover, various Deep RL algorithms that include Distributional and Parallel Computation approach do not run as expected on the Blender Python interpreter. In addition to that, the versioning of the python language in Blender is a bit different than to regular python. Hence, addition of new python module such as keras, tensorflow, pytorch is really a complex process.