How do I handle abusive ex-husband?

My ex-husband and I have joint custody of our 10 year old son. He has become more and more abusive in the past year, to the point our son says he doesn't want to go spend time with him, and sometimes begs not to have to go. Today my ex wanted me to drive and extra 45 mins (making an hour and a half drive for me) and only 30 minutes for him. I said "No, let s just stick with what we have always done." He became irate over the phone, cursing and screaming and stated, "I swear to F----ing God if I get within arms length of you ever again I will knock every tooth in your head down your f---ing throat!!!" And he screamed this to me in front of his new wife and our son. Now, I am afraid to meet him at all today, but I dont want to violate any written legal agreement that we have. Please help.