Indulging in only watching or listening to a movie requires more attention on one’s part, that is why very less people have even given it a thought.
On the other hand, at times this dependency on the audio-visual combo leads us to miss the film completely. Because as they say, perfect multi tasking is a myth and while watching a movie, we try to do the same by trying to focus both on audios and visuals at the same time.
Result: we end up watch the movie from the periphery missing the depth it may have.
I agree with you, to understand a movie completly and reach a level from where the maker is coming, watching it in one medium is completely important.
P.S. Not applicable for all films.