How do you know it's time to end a relationship?

I have been with the same guy for four years we have had our ups and downs ! In these years my ex has made it pretty rough he says it to help me but sometimes I think he’s just trying to hurt me more! He told me my current man is only using me he will get what he wants and leave me like he does everyone else! I never wanted to believe him but deep down I think I do! Like I told you we have been together for a minute now but during our relationship he had cheated I only know this to be true because he has had 3 kids and there not mine!!! His phone is always going off he says it’s baby mama bit I saw some messages from other chicks ! He tells me I’m his everything his ride or die! Is he ever going to just choose me? Is this something they will ever change? What do I do?