So far we have been dealing with a world where, following a certain policy, an agent takes certain actions, but acting in a vacuum state, with no interaction with the outside world.