I agree with you, for the most part.
I think the best approach, ideally, would be for library devs to offer compartmentalized versions of their library components.
For example, if you’re making a breaking change to component A of your library, but component B is what I use and no changes are made in this release, then give me the option of accessing your library sans part B.
If I’m not adopting functions from your API which are changed, yet my library (or app) breaks because of the update to said function(s) then there is a dependency problem in your library, or you haven’t properly outlined breaking changes.
A dream would be being able to do $npm install libraryname -breaks to be able to install minus breaking changes.
Of course, this would require the package dev to support compiling at install and a standardization of compartmentalized JavaScript libraries.
For example, if I could get JQuery 2.x minus [unused function here] by passing an npm argument, and pass such via my package.json, I could effectively parse the library for my app minus things I know will not be used or break my app, making it far more maintainable and potentially shrinking the deliverable size.
I think what we might need is a more robust package.json, npm and compatible libraries.
That said, if your library depends on another (and Swift is a bad example because it’s a language not a library) it’s a plugin, not a library.