Joint custody and residential custody question.?

My ex-husband and I share joint custody of our 10 year old son. He lived solely with me until he was 8. He wanted to spend time getting to know his Dad better, so I agreed to let him stay with him, even though it put him 5 hours away from me. That has been 2 years ago. He has spent all summers and holidays and birthdays with me, and now he says he wants to come back and live with me and my husband. Sometimes it is very difficult to get him to return to his father. He says he wants to finish this school year because of his friends, but when he comes to me for the summer he is not going back. As I said, our papers give us each joint custody of him, but he has residential custody per our agreement. I very much want him to come back and live with me but he father refuses to talk about it, and becomes very irate, curses and threatens everyone. Now, he wont answer the phone when I call to talk to our son. I got him a cellphone to work around this, but now his phone charger has mysteriously disappeared. My ex-husbands new wife says that our son tells her all the time that he wants to live with me, and that hi dad is trying to "brainwash" our son to stay with him. He calls me horrible names in front of him. My husband is a Master Sergeant in the USMC but he is also black. My exhusband says he will see me dead before he lets his son be raised by a "***--r". My ex is a coal miner in a small town where the schools are not even accredited, and has many anger managment issues. He pushes our son in pee-wee sports to the point that at 10 years old he already has bad ankle and neck. I know I can give our son a much better life, but I want to do this in a way that will have the least negative effects on our son. Please help!