My SO thinks I got with his brother: Thoughts?

So recently, my SO brought up something that we have not spoken about in years! So about 5+ years ago he “thinks” his brother and I had something going on. He’s brother went through a divorce and was staying with us for a couple of months because he did not have somewhere to go. He said one day when he came back from home; he heard loud/running noises coming from inside our home. (also we were living in a mobile home park at the time. when you are outside you can hear your neighbor’s steps/music/noise) and he said we were sitting in the living room watching Tv acting suspiciously. I don’t even remember that day, TBH. He never brought this topic up that same day or week, but a couple of months late. So, of course, I don’t even remember what day he is talking about. Within these last 5+ years, he has only brought this up a couple of times. Recently he brought it up again, but this time he brought up the suggestion of taking a police lie detector test! This seems outrageously ridiculous to me. Every time he brings this topic up, he makes me very angry and upset. We have been together for ten years, with 2 kids and in the way. I already explained to him that I have nothing to hide, and I’m not afraid to take a lie detector test. I’ve never done anything for my SO to believe I would ever cheat on him. Honestly, it was never pregnant at the moment I would have just packed my stuff and left. I am a SAHM and would be leaving empty-handed. I love this man such, but it hurts to get accused of such things. Are these lies detector stuff even accurate? This sounds like a joke to me. Like who thinks about stuff like this? This lie detector stuff sounds like something from a movie to me!!?!? Has anyone gone through anything like this?