Babysitting & my newborn advice

I’m needing some advice on what to do. My baby is due in August (not much longer) and her father and I are not together. He lives with his mother and her boyfriend. And I have my own apartment. In a nutshell I have told him since I got pregnant with our daughter that she wouldn’t live/stay at his moms house. That I didn’t want her there. And he’s always been slightly offended but never completely said that was an issue. His mom is a drug addict and so is he as well as her boyfriend. They have all kinds of people coming and going all the time and the house has roaches and rats. My child’s father has two other kids and he gets them all the time and they stay with him at his mothers house for visits. I don’t want my newborn daughter staying there with the random people and the drugs (that I’ve found left out more than once). Also we broke up when I found text messages on his phone to his other baby mama saying how he wants to take my daughter from me and raise her himself (with the help of his ex). She agreed with that whole scenario. Well it was recently brought up again about my daughter being at his moms house with him. He doesn’t currently work and hardly ever has a job. But I do. He said he wants to watch her and doesn’t want my sister to watch her while I go to work. And I said that’s fine but it will have to be at my apartment not at your moms. He didn’t take it well and said he’ll do whatever he wants with our daughter when he has her. More than once. I can’t trust him not to let random people hold and kiss her. I can’t trust him not to keep a constant eye on her (as she gets older and make sure she doesn’t get a hold of something she shouldn’t). I can’t trust him not to take off with her to his exes house. Or to leave her with his mom while he runs off to the casino (he’s done this so many times to his other kids). Also we used to live together and were together for 6 months before breaking up. I was pregnant when I moved out and we broke up. I told him if he can’t respect my wishes when it comes to our daughters safety and health then he won’t have anything to do with her at all. Because he won’t just get her and do what he wants with her. I don’t feel like I’m asking too much. And he’s had plenty of warning and time to get his own place so that this wouldn’t be an issue. I want the best for my girl. So my question is this, what do I do? Do I stick to my guns and make sure he knows I’m serious about this? He won’t have anything to do with her if he can’t respect my wishes about her safety. And by doing this I lose him (there’s still hope that we’ll be together again but his ex a problem) and my daughter loses her father unless he decides to change his mindset. Or do I get over it and allow my daughter to be in an unsafe environment with someone I don’t trust to take care of her properly when it comes to the things that are in said environment? My apartment is clean and it’s her home where she has everything she needs. Also keep in mind I’ve gotten her every single thing she needs and he hasn’t helped at all. Even tho I’ve asked. I have to pick him up so he can go to the appointments and be involved. And if he doesn’t watch her (he had a job until recently) the plans always been that my sister will. She’s a stay at home mom (and wonderful at it) and she’s very respectful of my wishes and respects me as a person and mother. Which I feel he does not do at all. He thinks I’m just being petty and doesn’t understand why I have a problem with my daughter being there. He was raised that way and is raising his other kids that way and it’s the way he lives. So in a way I understand why he doesn’t understand but my daughter is my everything and what’s best for her is my priority. I’m not due until august 17th and I have 6 weeks maternity leave before I go back to work and this is even a problem. What do I do?