How do you deal with family members who think your aspie just needs discipline?

My son says things that appear rude, but are really only his matter-of-factness. Family members get very nasty back to him, and obviously think he needs to be reprimanded. Since he is 18 yrs. old, he is expected to behave like an adult but he just does not get the social niceties, and family (especially his brother-in-law) get irate. I'm ready to stop having family get-togethers. What do I do?