[Paper Notes 3] Sim-to-Real Reinforcement Learning for Deformable Object Manipulation

1 The Motivation

There are rare papers tackling deformable object manipulation tasks via End-to-End Reinforcement Learning. Therefore, in this paper, they use a general-purpose deep reinforcement learning algorithm to tackle this problem and successfully solve three cloth manipulation tasks:1)Tape 2)Hanging 3)Diagonal folding from sim to real.

2 The Idea

This paper use an improved version of DDPG (with 7 extensions), which is very similar to Rainbow on discrete action space. It is hard to say novelty of this paper, but what they tried is a good practice for the community.

7 extensions for DDPG:

1. Prioritized Replay
2. N-step Returns
3. DDPGfD
4. Behavioral Cloning. Very similar idea with self-imitation learning
5. Reset to demonstration
6. TD3
7. Asymmetric Actor-Critic

3 What can we learn from this paper?

DDPG with extensions is a very powerful off-policy deep reinforcement learning algorithm for continuous control. Original DDPG performs notoriously unstable but it performs much better with so-many extensions.

We can apply this algorithm on many other robotic manipulation tasks or any continuous control tasks.

Besides of the algorithm, sim-to-real technique would become more popular since it works very well than what we believe.