I have not been on this website for a long time. Your story intrigued me because it sounds like my story. I work for a rather large firm in NYC. I do not think that my colleagues attack me as much as yours, but they do make up some situations for nothing. I ignore it and move on. It is about power and if you do not want power then don't worry. All these type of people want power over you. Sex is one way of controlling you, power? Yes.
Listen to all people above who posted good advice for you. Keep it clean, keep it low, keep it simple.