If the voice acting is anything to go by, it seems to be more of a "tone of voice isn't matched with what they're saying" thing.
I assume that they put some kind of emphasis on the opposite words, and everyone in town has been around them long enough to listen for it. Once they hear the difference compared to their normal words, they know it is the opposite.
(Minor spoilers for what she says if you marry her, but they're hilarious enough that I encourage you not to care about that. Also lots of Japanese, because apparently the undub comparisons are the only easily found voice clip dumps and .)