But you are not telling a story to a friend. You are not sitting next to them, able to gauge from their reactions if they understand what you are trying to impart or not. You have no chance to alter or expand on the fly to allow your listener to gain sufficient understanding of what might seem clear to you, in your mind, but is not clear to them.
This analogy fails on so many levels. In a novel, presumably written for the purpose of getting across the imagery as the writer envisioned it — be that a grand, detailed vision or a minimalist, sparse vision — to a reader with as much intact as possible, everything is description.
Action is description. Dialogue is description. Contemplation is description. The very way you write the dialogue describes the tone and attitudes of the characters, the nature of their conflict, the push-me-pull-you battle of understanding between them.
Attempting to dissuade the use of the most varied and powerful tool a writer possesses doesn’t strike me as helpful advice. Know the tool, know all its uses and purposes, its strengths and weaknesses, before asserting an opinion on it.