Is it wrong to uninvite someone from a baby shower?

Would it be wrong of me to univite someone to my baby shower? my moms boyfriend is the literal worst and we do not get along…i invited him to keep the peace but lately whenever my mom is with me visiting he blows her up wondering when she is going to be home which puts her in a bad mood…now i just feel weird being around him…idk what it is…what should i do