Tell him you want to swap, he goes back to work full time and you stay home. If he says no then tell him he needs to do his job which is looking after your child and during the night because you need to be able to work. If he says yes do to him what he has done to you. They need to learn that kids arent 9-5 with weekends off. Stay at home parent is hard work but really what is a break?
He does need a break i bet. I stay at home and my SO works and he doesn’t help at all really with kids. He might watch the older 2 while I take youngest to go shopping. But I’m exhausted 24/7 and he wonders why i just got to bed after kids fall asleep 
As someone who has been a stay at home mom as well as a working mom…
Take. Turns.
Take turns getting up with the baby. Take turns taking a weekend day off. If you don’t, you will end up hating one another. The appreciation is already gone.
He should be attending to the baby EVERYNIGHT on week days or tell him to GET HIS ASS TO WORK