I disagree. It's a mental association trick. If you say you don't like it when people do something, you are calling out a behavior. If you say you don't like it when men do something, you are (intentionally or not) causing the listener to associate men with that action when they may not have otherwise. You are proposing a negative association for them. It makes no difference how big the group is or how many subgroups there are.

There are definitely worse things that people can say about men than "I hate that some men do x", such as "all men are trash." (I've actually heard someone acting men should not be offended by this). I'm more offended by that sort of talk, but I think when people stop being careful about the words they choose, it will easily devolve into the latter.