Speaking for myself, more often than not, I suspect those who choose to characterize someone else as a "racist" actually have ulterior motives (which they themselves may not even be fully aware of) for doing so.
It's a last ditch effort to make someone swing to your point of view for fear of being labeled. You have no argument? Use the race/bigot/hater card.