My personal experience is probably far beyond yours since not only was I once a libertarian, I knew and still know many leading ones and was on the editorial board of several libertarian oriented publications even as I became more critical. I am still published as an respected critic by those very few libertarian publications, like the Molinari Review, that deal with anything other than spreading the party line.
And that saying about libertarians being right wingers is common because a great many people have the same experience I have had - under the talk of freedom is primarily a concern for freedom for me and lack of concern for freedom of others. One of their top theorists in the 70s even advocated the right for people desperate enough to sell themselves into slavery. That was Robert Nozick.
As I said, not all, but a great many.