What to do about my ex?

I have been separated/divorced 4 years.

My ex has two more kids with the lady he left me for (we both have made mistakes) and has three kids with me. We were high school sweethearts.
Still, 4 years later he won’t leave me alone. I’ve tried everything to get him to stop. In the beginning him and her would post horrible things in social media and it hurt a lot to see. But he would also act different when she wasn’t around. To this day it doesn’t hurt anymore and I just laugh it off when she posts drama that isn’t true and a lot of people are seeing the truth now. But she’s so blindsided by the way my ex is. There has been plenty of proof that he’s not good to her but he finds ways around blaming others and making himself look good and she believes him even with the proof in front of her face. He complains constantly to me about her. Tells me every time they fight and how he’s sick of telling her what he’s doing. (Part of me thinks she believes the truth but just can’t come to terms of it). Tells me he wants to leave but waiting for her to make an mistake so he can blame her for why he left. I told him he has to let go of me numerous times. He’s come to my house numerous times before work and tries for sex and I turn him down and well that’s makes for a very rough day of him being mean. (Claims he doesn’t get it from her nor wants it from her just me- if you think I believe that, trust me I don’t). It’s hard for me to find a relationship or date cause he’s always on my butt about it. Which I don’t care cause I still go on dates. But then I get constant calls and messages from him calling me a bad mom cause I leave my kids. Threats he’s taking me to court cause I pond my kids off on my mom (which my son has stayed one night with her in these last two months). I’m with them every single night. He barely even takes our youngest. (The other two just don’t care. They are older). When asked if he would it’s a fight. Then he makes excuses for the other two he has with her. So if I did something literally once a month, I’m a terrible parent. But part of me thinks that she’s part of the reasoning he doesn’t take our son just because of the constant posts about me. I try so hard for my kids and my youngest is starting to see what’s going on. He gets so mad and so upset when his dad promises to do things then ends up breaking the promise the day it’s suppose to happen.
So my thing is, I’ve called the cops on him numerous times about his threats and not leaving me alone. They tell him to stop. I don’t want to get a restraining order cause that’ll just hurt the kids even more and he’s not physically hurting me. Mentally maybe. But I just learned to brush it off.