Watching my daughters father other child

Hello. Long story short I am no longer with the father of my child. I have been with my partner now for six years we are in a same sex relationship. We have currently come to an issue because my daughter and her brother go to the same school so sometimes I will pick up my exes son because he got a call he’s always on call well now it’s becoming an issue and my partner is saying that I am very single minded that I am being treated like a doormat and that basically it can’t happen. Now I on the other hand I feel like we are one big blended family and I don’t see an issue with me watching my daughters brother, even though he is from another woman. My partner it insecure, but she is friends with my daughters father and I think that bothers me the most, being a little fake and hypocritical (I can’t be to close to him because he’s my ex, but she’s best friends with her ex of 8 years) I’m lost for words because I don’t have it in me to say I’m not gonna watch your child or tell my daughter we can’t watch your brother anymore. And I don’t think I’m going to, but now I’m going to have issues in my relationship I don’t know what to do. Am I wrong?