I do not get along with my husbands father: Advice?

My husband and I have been together for about 12 years and married for 8. We’ve never really had many arguments, but when we do, it’s usually the same topic which is his father. Bit of a background on him… He didn’t show up to our wedding and never gave us a reason why; he makes inappropriate/rude comments about me indirectly but knows I’m there; he grills my husband every time they speak about what he’s doing in his life etc. like he’s not good enough and he’s made no effort to contact or even try to get to know our son. We have a one-year-old, and due to covid, etc., we hadn’t made our trip down south (as we are usually expected to be the ones to travel); however, when borders opened, MIL and my family came to meet our son. Anywho, my concern is his lack of interest in knowing his only grandson or actually even making an effort with my husband. He is the only grandfather my child has as my dad passed away. However, he has made comments to others that we should be the ones to go to him and why he should bother coming here. He is retired, and my husband works 5-6 days a week in a job that at times he can’t be further than 30 mins away from the site, so traveling as a family is hard to do so. We are planning a trip home, and I don’t feel like my son, or I need that kind of negativity around us. If someone wants to be in his life, they will make an effort; he is the sweetest boy, I don’t believe we should force anyone to, and in this case, it happens to be the husband’s dad. I think my husband still holds on to that relationship because that’s what he is used to, or well, that’s the behavior he has grown up with, so he doesn’t really know much different. I suppose I am just looking for advice on what to do as I don’t want to see him, nor do I expect my child to have someone like that in his life? Or if anyone is in the same boat? Or how to approach this conversation with my husband? Or really what to do at all… Thank you so much