its my understanding (could be wrong) that only the java plugin2 support jvm arguments. On mac you need to enable java plugin2 by selecting out of process applets from the java control panel.
However for java plugin1 apple have backported some features to allow stuff like javafx to work, however I'm not sure if the jvm argument support was also backported, its unlikely it was.
By default (atm) safari and firefox use the java plugin1 (unless changed in the java control panel) and chrome uses (and only supports) java plugin2.
Apple have also recently crippled java performance on mac (when using java plugin2), this was done as they are trying to better integrate java applets with the web page, to allow other html stuff to overlap the applets they render the graphics offscreen and then draw it using their own CoreGraphics api. This does make the applets fit better but kills performance. You can read more about it in the change log here
, this change not only crippled java applets performance (including stuff like pulpcore) but also broke lwjgl and jogl applets.