Are we being extreme?

My 6 month old is teething so he's had fevers and has been very fussy, especially at night. It started on Sunday night and I was up with him nearly all night trying to console him. Well last night was the same, he cried nearly all night. No matter what I did or what I tried, he was still fussy and crying. Our living room is cooler than the rest of the house so I took him out there and walked around and bounced him in my arms. He calmed down and started to fall back to sleep so I put him in his bed that's next to my bed. He started crying almost immediately. I tried to see if he'd fall back to sleep and he didn't. At a loss of what to do and being exhausted from the night before and tending to my other 2 children during the day, I took him into the living room and laid him in his swing and a blanket and took a 15 minute breather to calm my nerves and try to relax enough to try again in a few minutes. In the time he was in his swing fussing my father whom we live with came out absolutely screaming. He yelled at the top of his lungs, "OH YEAH REAL GOOD PARENTING, LEAVING HIM OUT HERE TO CRY!!!!" It woke my other two children and my husband who are going back to school after winter break and work in the morning. My father's tv was BLARING LOUD the entire time I was out there trying to console my baby but when I sat him in his swing he quickly turned it off. Things have been going south for sometime now living with my father bc he's an alcoholic. While laid off we asked for some help with chores bc all our kids were sick and had to be taken to urgent care and he compared our request to slavery and said we demanded it and didn't even ask, I have video proof thanks to a nanny cam. We moved in under the impression that he wanted to be around his grandchildren and wanted to help us build credit but it was actually to babysit my father due to actions while under the influence of alcohol.

Anyways my husband and I have finally had enough of my father’s behavior and we’ve contacted a realtor. The environment has become so hostile that my friends who love coming over have sensed the change and I feel like it’s too toxic to raise my children around him. Are we being extreme? How would you respond?