Am I a part time mom?

My almost 5 year old son has been going every Friday to spend the night at his grandparents house (my husbands parents) because THEY want him to and my son absolutely loves it and looks forward to seeing his grandparents every weekend. My husband and I never asked them to do this… we don’t really go out and do much on the night that he’s away because my husband works long hours so I mostly just take the time to clean the house and sometimes when he gets off work we’ll go to dinner together or something. Well recently I was telling a friend about this because she and her boyfriend asked us to do something with them and if we would be bringing our son and so I told them that he usually spends the night at his grandparents every weekend but that I would ask my son if he’d like to go with us or go to his grandparents house… well when her boyfriend heard that he told me it was weird that I basically shared custody of my son and had to “ask permission” to do something with my son and that I was basically a part time mom… I was stunned because I never thought of it that way and we definitely don’t have to ask permission to do anything with our son but I just wanted to see what my child personally wanted to do… should I stop letting him go every single weekend and limit it? Is it weird that he goes there every single weekend?