vout_macosx: check whether the receiving window can respond to non-standard calls prior to calling them (close #6034)
(cherry picked from commit fe53c6cd)