Am I in the wrong?

I just got back from vacation. 5 days with my family and my best friends family. 4 adults and 5 kids. 3 girls and 2 boys. All 8 and under (8,7,5,4,3) . It took us 6 hours to drive back home, I told my friend she can go run errands and leave the girls with me at my house. My oldest 8 was super uncomfortable with wearing shorts and sitting for so long and had a irritation mark on his waist. He took his shorts of and just chilled like a normal day at our house. He had on tight boxer shorts. Nothing was poking out or peeping. My friend and her husband and their girls didn’t mind in the slightest. my husband told me today that a family member (who stopped by and saw my son in undies ) said it’s highly inappropriate and said we need to start wearing pants all day and get them used to always being clothes. “It’s got to start from the top” a direct dig at me. I walk around in underwear (full coverage) and a long ish shirt, unless we have company. Then I get dressed appropriately. As do my boys except this ONE time. Am I the AH for wanting them to mind their own?