vout_macosx: check whether the receiving window can respond to non-standard calls prior to calling them (close #6034)
(cherry picked from commit fe53c6cd32682d52f740bc49767c5eab68632e51)