the native window frame and widgets are used,
Actually I believe this is not entirely true (at least it is for some components). For example there is a well known MacOS Swing look and feel bug where it fails to render the JTree 100% correctly (I believe it's missing an animation), and pre-Vista versions of Java fail to render correctly on Vista and Win7 (I believe because they didn't fully support the new Windows look and feel). It's a copyright restriction that makes the various platform specific Swing look and feels platform specific.
So you might be able to hack the different platform specific look and feels to look greyed out. Making the close window action do nothing can be done without hacking it.
Although I agree with the others that you should not hack it. Whatever your 'no' or 'cancel' option is, have closing the dialog perform the same thing.