I flat out reject that white pride has to be about racism.
I agree that it does not "have to be" about racism. However, do you know any groups that, in practice, proclaim white pride and are not racist?
Rightly or wrongly, stigma does not derive from saying A has to mean some undesirable B, but from the simple existence of B whenever A is present.