Find myself agreeing with you rather alot! My workflow of late has tended to be stick to NPM package.json script entries (certainly) for smaller projects, especially those that are NodeJs centric, or don’t necessarily any more complex transpilation chaining etc.
If things genuinely become unwieldy (which hasn’t happened yet — you usually know upfront what sort of project size you’ll be working on, and what its build steps might look like), then its easy to introduce and lean on Webpack. Which sort of leaves Grunt and Gulp fallen between two stools.
Quick shout out to Marcus Hammarberg, whose Pluralsight course (link given above in article) redoubled my understanding of the various package.json script options, and how far you can take npm as the build tool of choice.