Dad Is Absent Yet Involved. Help!

I apologize for the length but I am at a loss on what to do. My son is 5. My son is also special needs (partially deaf, generalized anxiety diagnosis, global developmental delay). His father and I split in 2017 because he was cheating. Since this time his father has refused to pay child support, racking up tens of thousands in arrears, does not call my son, does not check in on my son, basically has nothing to do with him. Refuses to help with my sons uninsured medical bills, co payments, medical insurance or anything in regards to general support. Here is the kicker. My son is close to my ex’s mother, he spends time at her house 5-6x per year for 3 to 5 days. My ex goes to his mother’s to see my son, but has nothing to do with him otherwise. I am so tired of facilitating a half ass relationship between the two and being the 100% sole financial and day to day provider for my son. My ex reaps the benifits of seeing his son, with none of the work or responsibility!I am considering terminating visits (I have sole custody so this is all on my terms) until his father shows some involvement. The only issue being this will hurt my son in terms of seeing his grandmother. His grandmother refuses to deny her son access to my son when he is at her house.Advice?