I absolutely love worldbuilding as a hobby, and love to “explore” the worlds others have built. That’s part of the fun of reading imaginative fiction to me. And trying to fill in holes the writer left, or work out within the internal logic of an imaged world why things happened the way they did, is also part of the fun.
However, I 100% agree with you that realist worldbuilding is unncessary. If an author wants realist worldbuilding, then awesome. That’s one way to produce something beautiful. And if they don’t want to do that, then awesome, there are other good things in the world. And if I want to read a book not structured to respond to “worldbuilding”-style questions, but engage in fan-worldbuilding around it, that should be fine too: it’s a valid way I interact with the text that is fun for me. Maybe I will miss some of what the author intended; so I should strive to read as the author intended to be read. But when once I’ve done my best there, making a second pass wondering what the Death Star actually cost to build is a fun way to extend the entertainment value of the text.
Not sure if I’m agreeing or disagreeing; just a thought.