There is a lot more to the quote than the popular interpretation.
This teaching is addressed to Arjuna the warrior who wishes to flee the battle field rather than face the difficult task of fighting the opposing army consisting of his granduncle, his martial arts teacher and his cousins.
Krishna tells Arjuna ‘ Karmanye vadhikaraste Ma Phaleshu Kadachana’ — you have the qualification only to act (wage war) not renounce it. The instruction is to perform action as appropriate to his position (which for the warrior is to fight), without seeking benefits. There is no choice with regard discharge of duty.
Like Arjuna, we often get tempted to renounce the world, especially when we are going through a tough time in our life. But, if we retreat from the world without fully addressing our ego and our selfish desires, we will still be thinking about the material world in the confines of the cave or the ashram we have retreated into.