> That doesn’t change the fact that the people he was looking to punch in the face are by and large not Nazis.
Isn’t that his point? That he thought they were Nazis, because people were talking about how those people were Nazis, and THEY talked about Nazis, went around calling Hitler a genius and quoting Nazi propaganda and making anti-semitic jokes about wanting to destroy the Jews… But then he realized that, in fact, they were not Nazis at all, and thus, they didn’t need punching.
In other words, he wanted to attack Nazis. He believed that Nazis were here, and if that was the case, attacking them would be the appropriate response. However, he saw there were no Nazis here, and so, attacking them would be an inappropriate response to the situation.