I can't help but call them on it with a choice response in their language. I have also been known to just respond in English to whatever it is they've said... there is still that horrified shock when they realise I've understood them. I think it's very ill-mannered to switch to another language just to talk about someone as if they're not even there, and discomfiting rude people has never really troubled my conscience much. ;)