There really is nothing that one can do about that - one has to just let that go, because it is out of your control.
I trust that my friends are honorable and forthright, and make it clear that I am open to discuss any topic, including what an asshole I may or may not be. What else can be said, without at least knowing the remarks made about you?
Ruining the lives of others through rumor, conjecture, and implication are the hobbies of spiteful people who feel slighted (rightfully and otherwise). People have been doing it for as long as there have been people. Othello (and many other plays) is based on it.
All you can do is trust that you have chosen your friends wisely.