It seems pretty legitimate to worry about what will become of society if robots take over all or most unskilled labor jobs. It’s not hard to imagine that the unemployed and hungry would turn on the rich (and especially the rich who created the technology that put people out of work), though the Luddites, who found themselves in a similar situation, attacked machines and not people.
Still, I think it comes down to this: would you rather live in a richly-appointed prison for several decades, if not the rest of your life, or in a regular house — er — mansion? If the former, then go ahead and build your gold-plated bunker. If the latter, then perhaps get together with your billionaire friends and start a think tank to explore how to protect workers from tech-induced job loss and starvation, to lobby for policies that would help, etc. In this case, the right thing to do (helping people not starve because of something you invented) and your own self-interests (living above ground, not being known as one of the people who brought about the collapse of society, not taking the chance that someone will kill you before you can make it to your lushly-appointed missile silo) are aligned.