The unfortunate thing here is while your stance (if it's what you hold) isn't rooted in transphobia unfortunately in its consequences your stance enables transphobia. THat's the distinction here.
Let be come up with an analogy. The same argument could be made about banning Muslim immigration. Trump would say "I don't hate Muslims! I have tons of Muslim friends (and he does). I'm just worried about Muslims coming in from the Middle East and bombing us".
And that's a valid fear. However, and unfortunately, a policy banning an entire race from entering a country sets the stage for incredible amounts of racism. And that's only assuming it would work, despite the fact that it's so easy for terrorists to lie about their religion. SO that's how a policy can become bigoted even if the person voting for it isn't necessarily coming from a stance of bigotry. And unfortunately, if you support policies that are bigoted, that makes you no better than an actual bigot. I hope this makes sense.
And I think this will be very easily clarified as transgender people mingle more with other populations in North America, providing regular people with the opportunity to hear their stances.