My Sons Father Wants More Time With Him Than Custody Agreement

My sons father and I have a custody agreement. We went to mediation and agreed on a set fall schedule and then a summer schedule. The fall schedule is visitation for him every other weekend and one day a week (followed by one night every other week). The summer schedule is week by week with a night in the middle for the other parent. He recently asked me if I would agree to keep our summer schedule year round, and I told him I would think about it. The short answer is absolutely not. I do not want to be without my son for more time than we agreed to, but am worried that he will take me back to court if I don’t agree. My question is what are the odds he will actually be able to get what he is asking for, again he AGREED to the schedule, and I have never ever violated our order, I even go out of my way to give them more time with him when they have something going on that they want him included in during my time. I have been civilized and nice to them (he and his girlfriend) from the start, I just don’t want my time with my son cut any more than I already have to share.