Having 20,000 members is a very different thing than having 20,000 invisible and unorganized supporters. An organized Party can take serious coordinated action and have a long-term coordinated strategy; a bunch of people in their basements, even if they’re talking on the internet, cannot. For every Nazi Party member in Germany there were dozens who supported them strongly, hundreds who supported them tacitly, and thousands who were willing to tolerate them in power.

[For context, in 1923 Germany had about 60 million people. So I’m estimating hundreds of thousands of hardcore supporters, millions of general supporters, and tens of millions who were fine with them being in power.]

By your logic, we are also at deadly risk of being taken over by the Nation of Islam — and in fact, even more so, because NoI actually has a membership estimated in the 20,000–50,000 range (as of 2007), not just 20,000 hypothetical supporters hiding in basements across the country.