You’re right. Access shouldn’t be all or nothing. Luckily, this has changed in Android 6.0.

Starting Android 6.0, there is a better permission model. If the developer implemented this and you have Android 6.0 or higher, you get asked whenever the app needs the feature. You can also remove permissions afterwards from Settings.

In the Spotify case, it would only ask for permissions to see your photos at the time you want to change the playlist art. This makes it very clear what feature needs the permission, because you just started performing it.