I can't say I'm offended by it, but it can get irritating if you infer the other person has the wrong idea altogether. At work, for example, they all know I'm an uber-nerd. Heck, that was my nickname in business school, too, from a professor, because I was in the gaming industry. Everyone seems to take it in stride, but the minute you talk about costuming, the "perception flag" goes up...
So I refer to it as charity work all the time. I have a charity event tonight. It was a rough weekend of trooping for charity. We did a movie shoot for a really good cause.
And I refer to costuming in general as crafting, and use analogies people will understand based on their own interests, like knitting or woodworking. I see this as a crafting hobby, not so much a costuming hobby. "Costumes" implies certain things, right?
Try wordsmithing in advance.
Honestly I think cosplay at cons is considered fairly acceptable, like painting your face and waving flags at a sporting event. People might think it's weird, but I think the guys who paint themselves are weird, and that's okay.
But trooping for CHARITY, "bringing smiles to people's faces," and "sharing the love of star wars with those who need and appreciate it" is what it's really all about.