Kids playing outside

My kids play outside almost everyday with the other kids in the neighborhood they have all been friends for years and will play for hours. One of the kids has a pool and the kids will all swim but if my kids arent out at the time and they see the kids swimming they get their feelings hurt that no one told them about swimming. Would it be impolite if my kids asked if they could swim or should they always wait to be invited?