Should I go out with a guy that everyone else thinks is azz ?

Sure, but why does everyone else think he's a jerk? Why is he only being different with you and nobody else? Maybe he's not showing you his true colors and maybe he won't until the newness of the relationship wears off. Who knows.