Well, I thought the same and made the switch a couple of years ago. It was a Nexus 4 at the time. It was considered a Proper, capital P, Google Phone, if you remember (much more than Nexus 5, which was considered disappointing).
And it was good. Very good. But you know what: I switched back in less than two years.