Am I wrong for wanting my husband to get rid of his vacation home?

Can you please post anonymously? When I met my husband, he had just purchased a vacation home with his parents, right before meeting me. We are now married with 3 kids. He also has three other siblings who all have children, but they do not pay for the house at all because they did not go in when they bought it. His mom lets any and everybody use the house without asking us. They sleep in our bed, use our room, and it just bothers me. I find it to be unfair that we are paying half of the mortgage for a house, but his siblings all use it whenever they want or my inlaws friends free of charge. My husband would have never bought this with his parents if he had known we would be together and start a family so soon. He doesn’t want to upset anyone by backing out, so he just continues to pay for this house. It bothers me so much that money we could use for a lot of other things is going to a house we never even go to. My question is, what would you do, and how would you feel? Am I wrong for wanting to get rid of it?