Phone interference

Hi there. The weirdest thing has been going on with me while I'm on a phone. It only happens when I'm on a phone at one certain place and with one certain person. While in a middle of a conversation, I pause and don't say anything, the other person swears that he can hear me whisper, and he can hear the actual words, not just noice. My husband drives a long way home from work, and we lose connection sometimes, but if that was the case, we wouldn't be able to hear each other at all, or the call would just drop. He does talk to me via Bluetooth handsfree, but then again, how can he still be hearing me whispering some words when I'm not saying anything? It's just odd that it only happens when I'm on a phone with him and at home. Never had this issue with anyone else or while talking to him from some place else. And it wasn't just a one time thing, it has happened on multiple occasions. Has anyone have this issue before?