I get your frustration with Google PageSpeed. I guess everyone who has seriously tried to achieve a high PageSpeed number while further developing a site has experienced this. Costs a lot of time and money, and sometimes you wonder wether it was worth the effort.
After having spend many hours on many projects to optimize according to Google PageSpeed (and then I moved my own site to Medium, thus the drop) I would suggest you focus on user experience and the return of investment you would get from optimizing the whole website to achieve a high PageSpeed number. There are some things to optimize that give your visitors a quick and felt benefit and others are merely for the illusion of getting a higher rank in Google search. But there are many other which are very tricky to achieve, cost a lot in development effort, and do not high such a felt benefit compared to others.