Ex problems please help

I have been split with my child’s father for nearly 5 years. We split because he was hiding addiction, cheating and stole a large amount of money from me, which was the final straw and I ended it.
When we were together he wasn’t great, didn’t really get involved with his child, was majorly put out if I asked him to look after them while I was busy which was rare as he’d changed and I didn’t trust him alone with our child.
However since the spit we have had numerous issues, he’s turned up drunk and aggressive, he’s taken our child to pick up drugs, he’s hurt our child and didn’t take them to hospital, obviously I did as soon as I was aware of what happened and thankfully they got social services involved but they didn’t support me. I stopped our child going until I was confident that they would be safe. My ex lived with his parents at the time and it was agreed that he was to be supervised at all times unless it was a quick trip to the shops! And honestly that’s all they do together, and maybe a trip to the local park, nothing else!
My ex has also had a couple of unsuitable girlfriends in the past, they have also been drug users etc.
My dilemma is, my child doesn’t want to visit him, I’m really encouraging the access, try and jolly them along, but it’s becoming more difficult every time. My ex has a new girlfriend and my child said they’re planning on going to the zoo at the weekend! I want my child to have the best fun when they’re there, but my gut is in knots because I don’t tryst my ex for extended periods of time with our child, it was only last month he turned up drunk and crying at 4.30 in the afternoon to collect our child!
A good enough relationship for my child and their dad is so important to me as is their safety! But I’m in bits about this one!
Maybe I should add my child is 6, their grandparents and father are not loving, we are a very loving family, hugs and I love yous are the norm here. My child should visit their dad twice a week for dinner and every other weekend for a sleepover. My child also is very attached to me, which I am also struggling with keeping the very grey line between what’s their attachement to me and what’s them really not wanting to go!